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Abstract 


The  research  reported  here  concerns  the  principles 
used  to  automatically  generate  three-dimensional 
representations  from  line  drawings  of  scenes. 

The  computer  programs  Involved  look  at  scenes  which 
consist  of  pulyaedra  and  which  may  contain  shadows  and 
various  kinds  of  coincidentally  a. Igned  scene  features. 
Each  generated  description  includes  I nformat lonabout 
edge  shape  (convex^  concave/  occluding/  shadov;/  etc.)/ 
about  the  decomposition  of  the  scene  Into  bodieS/  about 
the  type  of  Illumination  for  each  region  (Illuminated/ 
projected  shadow/  or  oriented  away  from  the  light 
source)/  and  about  the  spaclal  orientation  of  regions. 
The  methods  used  are  based  on  the  labeling  schemes  of 
Huffman  and  Clowes;  this  research  provides  a 
considerable  extension  to  their  work  and  also  gives 
theoretical  explanations  to  the  heuristic  scene 
analysis  work  of  Guzman/  Winston/  and  others. 
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1.0  INTRODUCTION 

How  do  we  ascertain  the  shapes  of  unfamiliar  objects? 

Why  do  we  so  seldom  confuse  shadows  with  real  things?  How  do 
we  "favour  out"  shadows  when  looking  at  scenes?  How  are  we 
able  to  see  the  world  as  essentially  the  same  whether  It  Isa 
bright  sunny  day/  an  overcast  day/  or  a  night  with  only 
streetlights  for  Illumination?  In  the  terms  of  this  paper/ 
how  can  we  recognize  the  Identity  of  figures  1.1  and  1.2?  Do 
we  use  learning  and  knowledge  to  Interpret  what  V'/e  see/  or  do 
we  somehow  automatically  see  the  world  as  stable  and 
independent  of  lighting?  What  portions  of  scenes  can  we 
understand  from  local  features  alone/  and  what  configurations 
require  the  use  of  global  hypotheses? 

Various  theories  have  been  proposed  to  explain  how 
people  extract  three-dimensional  Information  from  scenes 
(Gibson  1950  is  an  excellent  reference).  It  Is  well  known 
that  we  get  depth  and  distance  Information  from  motion 
parallax  and,  for  objects  fairly  close  to  uS/  from  eye  focus 
feedback  and  parallax,  this  does  not  explain  how  we  are 

able  to  understand  the  three-dimensional  nature  of 
photographed  scenes.  Perhaps  we  acquire  knowledge  of  the 
shapes  of  objects  by  handling  them  and  moving  around  them/ 


FIGURE  1.1 


Reproduced  from 
best  available  copy. 
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and  use  rote  memory  to  assign  shape  to  those  objects  when  we 
recognize  them  In  scenes.  But  this  does  not  explain  how  we 
can  perceive  the  shapes  of  objects  we  have  never  seen  before. 
Similarly^  the  fact  that  we  can  tell  the  shapes  of  many 
objects  from  as  simple  a  representation  as  a  line  drawing 
shows  that  we  do  not  need  texture  or  other  fine  details  to 
ascertain  shape#  though  we  may  of  course  use  texture 
gradients  and  other  details  to  define  certain  edges. 

1  undertook  this  research  with  the  belief  that  It  Is 
possible  to  discover  rules  with  which  a  program  can  obtain  a 
three-dimensional  model  of  a  scene#  given  only  a  reasonably 
good  line  drawing  of  a  scene.  Such  a  program  might  have 
applications  both  In  practical  situations  and  In  developing 
better  theories  of  human  vision.  Introspectively#  I  do  not 
feel  that  there  Is  a  great  difference  between  seeing 
"reality”  and  seeing  line  drawings. 

Moreover#  there  are  considerable  difficulties  both  In 
processing  stereo  Images  (such  as  the  problem  of  deciding 
which  points  on  each  retina  correspond  to  the  same  scene 
point;  see  Guzman  1968#  Lerman  1970)  and  In  building  a  system 
Incorporating  hand-eye  coordination  which  could  be  •  sed  to 
help  explore  and  disambiguate  a  scene  (Gaschnig  1971).  It 
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seems  to  me  that  while  the  use  of  range  finders,  multiple 
light  sources  to  help  eliminate  shadows  (Shirai  1071),  and 
the  restriction  of  scenes  to  known  objects  may  all  prove 
useful  for  practical  robots,  these  approaches  avoid  camlng  to 
grips  with  the  nature  of  human  perception  vis-a-vis  the 
implicit  three-dimensional  Information  In  line  drawings  of 
real  scenes.  While  I  would  be  very  cautious  about  claiming 
parallels  between  the  rules  in  my  pv ogram  and  human  visual 
processes,  at  the  very  least  I  have  demonstrated  a  number  of 
capable  vision  programs  which  require  only  fixed,  monocular 
line  drawings  for  their  operation. 

In  this  thesis  I  der-cribe  a  working  collection  of 
computer  programs  which  reconstruct  three-dimensional 
dfc.  •’.  I'stions  from  line  drawings  which  are  obtained  from 
scenes  composed  of  plane-faced  objects  under  various  lighting 
conditions.  In  this  description  the  system  identifies  shadow 
lines  and  regions,  groups  regions  whir*  i. elong  to  the  same 
object,  and  notices  such  relations  as  contact  or  lack  of 
contact  between  the  objects,  support  and  in-f ront-of/behl nd 
relations  between  the  objects  as  well  as  Information  about 
the  spv^cial  orientation  of  various  regions,  all  using  the 
description  it  has  generated. 
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1.1  DESCRIPTIONS 

The  overall  goal  of  the  system  Isto  provide  a  precise 
description  of  a  plausible  scene  which  could  give  rise  to  a 
particular  line  drawing.  It  is  therefore  Important  to  have  a 
good  language  in  which  to  describe  features  of  scenes.  Since 
I  wish  to  have  the  program  operate  on  unfamiliar  objects/  the 
language  must  be  capable  of  describing  such  objects.  The 
language  I  have  used  is  an  expansion  of  the  labeling  system 
developed  by  Huffman  (Huffman  1971)  In  the  United  States  and 
Clowes  (Clowes  1971)  In  Great  Britain. 

The  language  employs  labels  which  are  assigned  to  line 
segments  and  regions  in  the  scene.  These  labels  describe  the 
edge  geometry/  the  connection  or  lack  of  connection  between 
adjacent  regions/  the  orientation  of  each  region  in  three 
dimensions/  and  the  nature  of  the  illumination  for  each 
region  (illuminated/  projected  shadow  region/  or  region 
facing  away  from  the  light  source).  The  goal  of  the  program 
is  to  assign  a  single  label  value  to  each  line  and  region  in 
the  line  drawing/  except  In  cases  where  humans  also  find  a 
feature  to  be  ambiguous. 
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This  language  allows  precise  definitions  of  such 
concepts  as  supported  by#  In  front  of#  behind#  rests  against# 
shadows#  is  shadowed  by#  ts  capable  of  supporting#  leans  on# 
and  others.  Thus#  if  it  is  possible  to  label  each  feature  of 
a  scene  uniquely#  then  it  Is  possible  to  directly  extract 
these  relations  from  the  description  of  the  scene  based  on 
this  labeling. 


1.2  JUNCTION  LABELS 


Much  of  the  program’s  power  is  based  on  access  to  K^its 
of  possible  line  label  assignments  for  each  type  of junction 
in  a  line  drawing.  While  a  natural  language  analogy  to  these 
labels  could  be  misleading#  I  think  that  it  helps  in 
explaining  the  basic  operation  of  this  portion  of  the 
program. 


if  we  think  of  each  possible  label  for  a  line  as  a 
letter  in  the  alphabet#  then  each  junction  must  be 
labeled  with  an  ordered  list  of  "letters"  to  form  a 
legal  "word"  in  the  language.  Thus  each  "word" 
represents  a  physically  possible  Interpretation  for  a 
given  junction.  Furthermore*  each  "word"  must  match  the 
"words"  for  surrounding  junctions  in  order  to  form  a 
legal  "phrase"#  and  all  "phrases"  !n  the  scene  must 
agree  to  form  a  legal  "sentence"  for  the  entire  scene. 
The  knowledge  of  the  sys*’em  is  contained  In  (1;  a 
dictionary  made  up  of  every  leial  "word"  for ®ch  type 
of  junction#  and  (2)  rules  by  which  "words"  can  legally 
combine  with  other  "words".  The  range  of  the  dictionary 
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entries  defines  the  universe  of  the  program;  this 
universe  can  be  expanded  by  adding  new  entries 
systematically  to  the  dictionary. 

In  fact^  the  "dictionary'*  need  not  be  a  stored  list. 

The  dictionary  can  consist  of  a  relatively  small  list  of 
possible  edge  geometries  for  each  junction  type/  and  a  set  of 
rules  which  can  be  used  to  generate  the  complete  dictionary 
from  the  original  lists.  Depending  on  the  amount  of  computer 
memory  available/  It  nay  either  be  desirable  to  store  the 
complete  lists  as  compiled  knowledge  or  to  generate  the  lists 
when  they  are  needed.  In  my  current  program  the  lists  are 
for  the  most  part  precompi led. 

The  composition  of  the  dictionary  Islntcrestlng  In  Its 
own  right.  While  some  basic  edge  geometries  give  rise  to 
many  dictionary  entries/  some  give  rise  to  very  few.  The 
total  number  of  entries  sharing  the  same  edge  geometry  can  be 
as  low  as  three  for  some  ARROW  junctions/  Including  shadow 
edgeS/  while  the  number  generated  by  some  FORK  junction  edge 
geometries  Is  over  270/000  (Including  region  orientation  and 
illumination  values). 


s 
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1.3  JUNCTION  LABEL  ASSIGNMENT 

There  Is  a  considerable  amount  of  local  Information 
which  can  be  used  to  select  a  subset  of  the  total  number  of 
dictionary  entries  which  are  consistent  with  a  particular 
Junction.  The  first  piece  of  Information  Isal ready  Included 
Implicitly  In  the  Idea  of  junction  type.  Junctions  are  typed 
according  to  the  number  of  lines  which  make  up  the  Junction 
and  the  two  dimensional  arrangement  of  these  lines.  Figure 
1.3  shows  all  the  Junction  types  which  can  occur  In  the 
universe  of  the  program.  The  dictionary  Is  arranged  by 
Junction  type,  and  a  standard  ordering  Is  assigned  to  all  the 
line  segments  which  make  up  Junctions  (except  FORKS  and 
MULTIS). 

The  program  can  also  use  local  region  brightness  and 
line  segment  direction  to  preclude  the  assignment  of  certain 
labels  to  lines.  For  example.  If  It  knows  that  one  region  Is 
brighter  than  an  adjacent  region,  then  the  line  which 
separates  the  regions  can  be  labeled  as  a  shadowreglon  In 
only  one  way.  There  are  other  rules  which  relate  region 
orientation,  light  placement  and  region  Illumination  as  well 
as  rules  which  limit  the  number  of  labels  which  can  be 
assigned  to  line  segments  which  border  the  support  surface 
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for  the  scene.  The  program  Is  able  to  combine  all  these 
types  of  Information  In  finding  a  list  of  appropriate  labels 
for  a  single  junction. 

1.4  COMBINATION  RULES 

Combination  rules  are  used  to  select  from  the  Initial 
assignments  the  label#  or  labels#  which  correctly  describe 
the  scene  features  that  could  have  produced  each  junction  In 
the  given  line  drawing.  The  simplest  type  of  combination 
rule  merely  states  that  a  label  Is  a  possible  description  for 
a  junction  If  and  only  If  there  Is  at  least  one  label  which 
"matches"  It  assigned  to  each  adjacent  junction.  Two 
junction  labels  "match"  If  and  only  If  the  line  segment  which 
joins  the  junctions  gets  the  same  Interpretation  from  both  of 
the  junctions  at  Its  ends. 

Of  course,  each  Interpretation  (line  label)  Is  really  a 
shorthand  code  for  a  number  cf  properties  of  the  line  and  Its 
adjoinin'-  regions.  If  the  program  can  show  that  any  one  of 
these  constituent  values  cannot  occur  In  the  given  scene 
context,  then  the  whole  complex  of  values  for  that  line 
expressed  Implicitly  in  the  interpretation  cannot  he  possible 
either  and,  furthermore,  any  junction  label  which  assigns 
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this  interpretation  to  the  line  segment  can  be  eliminated  as 
well.  Thus^  when  it  chooses  a  label  to  describe  a 
particular  junction/  it  constrains  all  the  junctions  which 
surround  the  regions  touching  this  junction/  even  though  the 
combination  rules  only  compare  adjacent  junctions. 

More  complicated  rules  are  needed  If  it  is  necessary  to 
relate  junctions  which  do  not  share  a  visible  region  or  line 
segment.  For  example/  <  thought  at  the  outset  of  my  work 
that  it  might  be  necessary  to  construct  models  of  hidden 
vertices  or  features  which  faced  away  from  the  eye  in  order 
to  find  unique  labels  for  the  visible  features.  The 
difficulty  in  this  is  that  unless  a  program  can  find  which 
lines  represent  obscuring  edges/  it  cannot  know  where  to 
construct  hidden  features/  but  If  it  needs  the  hidden 
features  to  label  the  lineS/  i tmay  not  be  able  to  decide 
which  lines  represent  obscuring  edges.  As  it  turns  out/  no 
such  complicated  rules  and  constructions  are  necessary  in 
general;  most  of  the  labeling  problem  can  be  solved  by  a 
scheme  which  only  compares  adjacent  junctions. 
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1.5  EXPE^IIMENTAL  RESULTS 

When  I  began  to  write  a  program  to  Implement  the  system 
I  had  devised^  I  expected  to  use  a  tree  search  system  to  find 
which  labels  or  "words”  could  be  assigned  to  each  junction. 
However,  the  number  of  dictionary  entries  for  each  type  of 
junction  Is  very  high,  (there  are  almost  3000  different  ways 
to  label  a  FORK  junction  before  even  considering  the  possible 
region  orientations!)  so  I  decided  to  use  a  sort  of 
"filtering  program"  before  doing  a  full  tree  search. 

The  program  computes  the  full  list  of  dictionary  entries 
for  each  junction  In  the  scene,  eliminates  from  the  list 
those  labels  which  can  be  precluded  on  the  basis  of  local 
features,  assigns  each  reduced  list  to  Its  junction,  and  then 
the  filtering  program  computes  the  possible  labels  for  each 
line,  using  the  fact  that  a  line  label  Is  possible  If  and 
only  if  there  is  at  least  one  junction  label  at  each  end  of 
the  line  which  contains  the  line  label.  Thus,  the  list  of 
possible  labels  for  a  line  segment  Is  the  intersection  of  the 
two  lists  of  possibilities  computed  from  the  junction  labels 
at  the  ends  of  the  line  segment,  if  any  junction  label  would 
assign  a  Interpretation  to  the  line  segment  which  Is  not  In 
this  intersection  list,  then  that  label  can  be  eliminated 
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from  consideration.  The  filtering  program  uses  a  network 
iteration  scheme  to  systematically  remove  all  the 
Interpretations  which  are  precluded  by  the  elimination  of 
labels  at  a  particular  junction. 

When  !  ran  this  filtering  program  1  was  amazed  to  find 
that  In  the  first  few  scenes  I  trled^  this  program  found  a 
unique  label  for  each  line.  Even  when  I  tried  considerably 
more  complicated  scenes#  there  were  only  a  few  lines  In 
general  which  were  not  uniquely  specified#  and  some  of  tf.ise 
were  essentially  ambiguous#  I.e.  I  could  not  decide  exactly 
what  sort  of  edge  gave  rise  to  the  line  segment  myself.  The 
other  ambiguities#  I.e.  the  ones  volch  I  could  resolve 
myself#  In  general  require  that  the  program  recognize  lines 
which  are  parallel  or  coll  Inear  or  regions  which  meet  along 
more  than  one  line  segment#  and  hence  require  more  global 
agreement. 

I  have  been  able  to  use  this  system  to  Investigate  a 
large  number  of  line  drawings#  Including  ones  with  missing 
lines  and  ones  with  numerous  accidentally  aligned  junctions. 
From  these  investigations  I  can  say  with  some  certainty  which 
types  of  scene  features  car.  'e  handled  by  the  filtering 
program  and  which  require  more  complicated  processing. 
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Whether  or  not  more  processing  Is  required,  the  filtering 
system  provides  a  computationally  cheap  method  for  acquiring 
a  great  deal  of  information.  For  enample.  In  most  scenes  a 
large  percentage  of  the  line  segments  are  unambiguously 
labeled,  and  more  complicated  processing  can  be  directed  to 
the  areas  which  remain  ambiguous.  As  another  example,  if  I 
only  wist)  to  know  which  lines  are  shadows  or  which  lines  are 
the  outside  edges  of  objects  or  how  many  objects  there  are  In 
the  scene,  the  program  may  be  able  to  get  this  information 
even  though  some  ambiguities  remain,  since  the  ambiguity  may 
only  involve  region  Illumination  type  or  region  orientation. 

Figure  1.4  shows  some  of  the  scenes  which  the  program  Is 
able  to  handle.  The  segments  which  remain  ambiguous  after 
its  operation  are  marked  with  stars,  and  the  approximate 
amount  of  time  the  program  requires  to  label  each  scene  is 
marked  below  it.  The  computer  is  a  PDP>10,  and  the  program 
Is  written  partially  in  MICRO-PLANNER  (Sussman  et  al  1971) 
and  partially  in  compiled  LISP. 
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1.6  COMPARISON  WITH  OTHER  VISION  PROGRAMS 


My  system  differs  from  previously  proposed  ones  In 
several  Important  ways; 


First/  It  is  able  to  handle  a  much  broader  range  of 
scene  types  than  have  previous  programs.  The  program 
"understands"  shadows/  some  junctions  which  have  missing 
IlneS/  and  apparent  alignment  of  edges  caused  by  the 
particular  placement  of  the  eye  with  respect  to  the  scene/  so 
that  no  special  effort  needs  to  be  made  to  avoid  problematic 
features. 


Second/  the  design  of  the  program  facilitates  its 
Integration  with  line-finding  programs  and  higher-level 
programs  such  as  programs  which  deal  with  natural  language  or 
overall  system  goals.  The  system  can  be  used  to  write  a 
program  which  automatically  requests  and  uses  many  diff  rent 
types  of  information  to  find  the  possible  interpretations  for 
a  single  feature  or  portion  of  a  scene. 

Third/  the  program  is  able  to  deal  with  ambiguity  In  a 
natural  manner.  Some  features  In  a  scene  can  be  ambiguous  to 
a  person  looking  at  the  same  scene  and  the  program  preserves 
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these  various  possibl t ties.  This  tolerance  for  ambiguity  Is 
central  to  the  philosophy  of  the  program;  rather  than  trying 
to  pick  the  "most  probable"  Interpretation  of  any  features/ 
the  program  operates  by  trying  to  eliminate  Impossible 
Interpretations.  If  it  has  been  given  Insufficient 
information  to  decide  on  a  unique  possibility/  then  It 
preserves  all  the  active  possibilities  It  knows.  Of  course 
if  a  single  interpretation  Is  required  for  some  reason/  one 
can  be  chosen  from  this  list  by  heuristic  rules. 

Fourth/  the  program  Is  algorithmic  and  does  not  require 
facilities  for  back-up  If  the  filter  program  finds  an 
adequate  description.  Heuristics  have  been  used  In  all 
previous  vision  programs  to  approximate  reality  by  the  most 
likely  interpretation.  This  may  simplify  some  problems/  but 
sophisticated  programs  are  needed  to  patch  up  the  cases  where 
the  approximation  is  wrong;  In  my  program  I  have  used  as 
complete  a  description  as  I  could  devise  with  the  result  that 
the  programs  are  particularly  simple/  transparent  and 
powerful . 

Fifth/  because  of  this  simplicity/  I  have  been  able  to 
write  a  program  which  operates  very  rapidly.  As  a  practical 
matter  this  is  very  useful  for  debugging  the  sysvom/  and 
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allows  modifications  to  be  made  with  relative  ease. 

Moreover^  because  of  Its  speedy  I  have  been  able  to  test  the 
program  on  many  sep<^rate  line  drawings  and  have  thus  been 
able  to  gain  a  clearer  understanding  of  the  capabilities  and 
ultimate  limitations  of  the  program.  In  turn^  this 
understanding  has  led  and  should  continue  to  lead  to  useful 
modifications  and  a  greater  understanding  of  the  nature  and 
complexity  of  procedures  necessary  to  handle  various  types  of 
scene  features. 

Sixth,  as  explained  In  the  next  section,  the  descriptive 
language  provides  a  theoretical  foundation  of  considerable 
value  In  explaining  previous  work. 

1.7  HISTORICAL  PERSPECTIVE 

One  of  the  great  values  of  the  extensive  descriptive 
apparatus  I  have  developed  Is  Its  ability  to  explain  the 
nature  and  shortcomings  of  past  work.  I  will  discuss  In 
Chapter  9  how  my  system  helps  In  understanding  the  work  of 
Guzman  (Guzman  1968),  Rattner  (Rattner  1970),  Huffman 
(Huffman  1971),  Clowes  (Clowes  1971),  and  Orban  (Orban  1970); 
and  to  explain  portions  of  the  work  of  Winston  (Winston  1970) 
and  Flnln  (Flnln  1971a,  1971b).  For  example,  I  show  how 
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various  concepts  such  as  support  can  be  formalized  In  my 
descriptive  language.  From  this  historical  comparison 
emerges  a  striking  demonstration  of  the  ability  of  good 
descriptions  to  both  broaden  the  range  of  applicability  of  a 
program^  and  simplify  the  program  structure. 

1.8  iMPLt CAT  IONS  FOR  HUMAN  PERCEPTION 

My  belief  that  the  rules  which  govern  the  interpretation 
of  a  line  drawing  should  be  simple  is  based  on  the  subjective 
impression  that  little  abstraction  o*-  processing  of  any  type 
seems  to  be  required  for  me  to  be  able  to  recognize  the 
shadows/  object  edges/  etc.  In  such  a  drawing/  in  cases  where 
the  drawing  is  reasonably  simple  and  complete,  t  do  not 
believe  that  human  perceptual  processes  necessarily  resemble 
the  processes  in  my  program/  but  there  are  various  aspects  of 
my  solution  which  appeal  to  my  Intuition  about  the  nature  of 
that  portion  of  the  problem  which  is  independent  of  the  type 
of  perceiver.  I  think  It  is  significant  that  my  program  is 
as  simple  as  it  iS/  and  that  the  information  stored  in  it  is 
so  independent  of  particular  objects.  Back-up  is  not 
necessary  in  general;  the  system  works  for  picture  fragments 
as  well  as  for  entire  scenes;  the  processing  time  required 
is  proportional  to  the  number  of  line  segments  and  not  an 
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exponential  function  of  the  number;  all  these  facts  lead  me 
to  believe  that  my  research  has  been  in  the  right  directions. 

Clearly  there  are  considerable  obstacles  to  be  overcome 
in  extending  this  work  to  general  scenes.  For  simple  curved 
objects  such  as  cylinders^  spheres^  cones^  and  conic 
sections,  there  should  be  no  particular  problem  in  using  the 
type  of  program  I  have  written.  (For  a  quite  different 
approach  to  the  handling  of  curved  objects,  see  Horn  1970.) 
i  also  believe  that  it  will  be  possible  to  handle  somewhat 
more  general  scenes  (for  instance  scenes  containing 
furniture,  tools  and  household  articles)  by  approximating  the 
objects  in  them  by  simplified  "envelopes"  which  preserve  the 
gross  form  of  the  objects  yet  which  can  be  described  In  terms 
like  those  I  have  used.  In  my  estimation  such  processing 
cannot  be  done  successfully  until  the  problem  of 
reconstructing  the  invisible  portions  of  the  scene  is  solved. 
This  problem  is  intimately  connected  with  the  problem  of 
using  the  stored  description  of  an  object  to  guide  the  search 
for  instances  of  this  object,  or  similar  objects  in  a  scene. 
The  ability  to  label  a  line  drawing  in  the  manner  I  describe 
greatly  simplifies  the  specification  and  hopefully  will 
simplify  the  solution  of  these  problems.  Chapter  8  deals 
with  natural  extensions  of  my  program  which  I  believe  will 
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2.0  QUICK  SYNOPSIS 

This  chapter  provides  a  quick  look  at  some  of  the 
technical  aspects  of  ny  work.  All  topics  covered  here  are 
treated  either  In  greater  detail  or  from  a  different 
perspective  In  later  chapters.  For  a  hurried  reader  this 
chapter  provides  a  map  to  the  rest  of  the  paper/  and  enough 
background  to  understand  a  later  chapter  without  reading  all 
the  intervening  ones. 

2.1  THE  PROBLEM 

In  what  follows  I  frequently  make  a  distinction  between 
the  scene  Itself  (objects/  table/  and  shadows)  and  the 
retinal  representation  of  the  scene  as  a  two-dimensional  line 
drawing.  I  will  use  the  terms  vertex/  edge  and  surface  to 
refer  to  the  scene  features  which  map  into  junction/  line  and 
region  respectively  In  the  line  drawing. 

Our  first  subproblem  is  to  develop  a  language  that 
allows  us  to  relate  these  two  worlds.  I  have  done  this  by 
assigning  names  called  labels  to  lines  in  the  line  drawing/ 
after  the  manner  of  Huffman  (Huffman  1971)  and  Clowes  (Clowes 
1971),  ThuS/  for  example/  In  figure  2.1  line  segment  J1-J2 
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Is  labeled  as  a  shadow  edge^  tine  J2'’J3  Is  labeled  as  a 
concave  edge^  line  J3-dl4  Is  labeled  as  a  convex  edge^  line 
J4'J5  Is  labeled  as  an  obscuring  edge  and  line  tll2-tJ13  Is 
labeled  as  a  crack  edge.  Thus#  these  terms  are  attached  to 
parts  of  the  drawing/  but  they  designate  the  kinds  of  things 
found  in  the  three-dimensional  scene. 

When  we  look  at  a  line  drawing  of  this  sort/  we  usually 
can  easily  understand  what  the  line  drawing  represents.  In 
terms  of  a  labeling  scheme  either  (1)  we  are  able  to  assign 
labels  uniquely  to  each  line/  or  <2)  we  can  say  that  no  such 
scene  could  exist/  or  (3)  we  can  say  that  although  It  Is 
impossible  to  decide  unambiguously  what  the  label  of  an  edge 
should  be/  It  must  be  labeled  with  one  member  of  some 
specified  subset  of  the  total  number  of  labels.  What 
knowledge  Is  needed  to  enable  the  program  to  reproduce  such 
labeling  assignments? 

Huffman  and  Clowes  provided  a  partial  answer  In  their 
papers.  They  pointed  out  that  each  type  of  junction  can  only 
be  labeled  In  a  few  wayS/  and  that  if  we  can  say  with 
certainty  what  the  label  of  one  particular  line  IS/  we  can 
greatly  constrain  all  other  lines  which  Intersect  that  line 
segment  at  Its  ends.  As  a  specific  example/  If  one  branch  of 
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an  L  junction  is  labeled  as  a  shadow  edge^  then  the  other 
branch  must  be  labeled  as  a  shadow  edge  as  well. 

Moreover/  shadows  are  directional/  I.e.  In  order  to 

specify  a  shadow  edge/  It  must  not  only  be  labeled  "shadow" 

but  must  also  be  marked  to  Indicate  which  side  of  the  edge  Is 

shadowed  and  which  side  Is  Illuminated.  Therefore/  not  only 

the  type  of  edge  but  the  nature  of  the  regions  on  each  side 
\ 

can  be  constrained. 

These  facts  can  be  Illustrated  In  a  jigsaw  puzzle 
analogy/  shown  In  figure  2.2.  Given  the  five  different  edge 
types  I  have  discussed  so  far,  there  are  seven  different  ways 
to  label  any  line  segment.  This  Implies  that  If  all  line 
labels  could  be  assigned  independently  there  would  be  7^  ■  49 
different  ways  to  label  an  I,  7®  ■  343  ways  to  label  a 
three-line  junction/  etc.  In  fact  there  are  only  9  ways  In 
which  real  scene  features  esn  map  Into  Us  on  a  retinal 
projection.  Table  2.1  summarizes  the  ways  In  which  junctions 
can  be  assigned  labelings  from  this  set.  In  figure  2.3/  I 
show  all  the  possible  labelings  for  each  junction  type/ 
limiting  myself  to  vertices  which  are  formed  by  no  more  than 
three  planes  (trihedral  vertices)  and  to  junctions  of  five  or 
fewer  lines.  In  Chapter  3  I  explain  how  to  obtain  the 
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junctions  in  figure  2.3;  I  do  not  expect  that  It  should  be 
obvious  to  you  how  one  could  obtain  these  junctions,  in 
general/  for  clarity#  I  have  tried  to  use  the  word  labeling 
to  refer  to  the  simultaneous  assignment  of  a  number  of  line 
labels.  Labels  thus  refer  to  line  interpretations#  and 
labelings  refer  to  junction  or  scene  interpretations. 


2.2  SOLVING  THE  LABEL  ASSIGNMENT  PROBLEM 


Labels  can  be  assigned  to  each  line  segment  by  a  tree 
search  procedure,  in  terms  of  the  jigsaw  puzzle  analogy# 
imagine  that  we  have  the  following  Items: 


1.  A  board  with  channels  cut  to  represent  the  line 
drawing;  the  board  space  can  accept  only  L  pieces  at  each 
place  where  the  line  drawing  has  an  L#  only  ARROW  pieces 
where  the  line  drawing  has  an  ARROW#  etc.  Next  to  each 
junction  are  three  bins#  marked  "junction  number"#  "untried 
labels"#  and  "tried  labels". 

2.  A  full  set  of  pieces  for  every  space  on  the  board.  If 
the  line  drawing  represented  by  the  board  has  five  Ls  then 
there  are  five  full  sets  of  L  pieces  with  nine  pieces  in  each 
set. 

3.  A  set  of  junction  number  tags  marked  Jl#  J2#  J3# 

...#  dn#  where  n  Is  the  number  of  junctions  on  the  board. 

4.  A  counter  which  can  be  set  to  any  number  between  1 
and  n. 


The  tree  search  procedure  can  then  be  visualized  as 
fol lows? 


SECTION  2.2  40 


Step  1;  Name  each  junction  by  placinz  a  junction  number  tag 
In  each  bln  marked  "junction  number". 

Step  2:  Place  a  full  set  of  the  appropriate  type  of  pieces  In 
the  "untried  labels"  bln  of  each  junction. 

Step  3:  Set  the  counter  to  1.  From  here  on  In  Nc  will  be 
used  to  refer  to  the  current  value  of  the  counter.  Thus  If 
the  counter  is  set  to  6/  then  J(Nc)  •  6. 

Step  4;  Try  to  place  the  top  piece  from  the  "untried  labels" 
bin  of  junction  J(Nc)  In  board  space  J(Nc).  There  are 
several  possible  outcomes: 

4A.  If  the  piece  can  be  placed  (I.e.  the  piece  matches 
all  adjacent  pieces  already  placed^  If  any}^  then 

Al.  If  Nc  <  n#  Increase  the  counter  by  one  and 
repeat  Step  4. 

A2.  If  Nc  •  n#  then  the  pieces  now  on  the  board 
represent  one  possible  labeling  for  the  line  drawing.  If 
this  Is  true  then 

I,  Write  dov/n  or  otherwise  remember  the 

labeling,  and 

II.  Trans<-er  the  piece  In  space  n  back  Into  the 
n-th  "untried  labels"  bln,  and 

ill.  Go  to  Step  5. 

4B.  If  the  piece  cannot  be  placed,  put  It  in  the  "tried 
labels"  bln  and  repeat  Step  4. 

4C.  If  there  are  no  more  pieces  In  the  "untried  labels" 
bln,  then 

C2,  If  Nc  ■  1,  we  have  found  all  (If  any)  possible 
labelings,  and  the  procedure  Is  DONE. 

C2.  Otherwise,  go  to  Step  5. 

Step  5:  Do  all  the  following  steps: 
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I,  Transfer  all  the  pieces  from  the  Nc-th 
‘'tried  labels"  bln  into  the  Nc-th  “untried  labels"  bln#  and 

51,  Transfer  the  piece  In  space  Nc-1  into  Its 
"tried  labels"  bln#  and 

III.  Set  Che  counter  to  Nc-1/  and  go  to  Step  4. 

To  see  how  this  procedure  works  In  practice/  see  figure 
2.4.  ^or  this  example  assume  that  the  pieces  are  piled  so 
that  the  order  in  which  they  are  tried  Is  the  same  as  the 
order  In  which  the  pieces  are  listed  In  figure  2.3.  The 
example  Is  carried  out  only  as  far  as  the  first  labeling 
obtained  by  the  procedure.  There  Is/  of  course/  at  least  one 
other  labeling/  namely  the  one  we  could  assign  by  inspection. 
The  "false"  labeling  found  first  could  be  eliminated  in  this 
case  by  a  program  If  It  knew  that  R3  Is  brighter  than  R1  or 
that  R2  is  brighter  than  Rl.  It  could  then  use  heuristics 
which  only  allow  it  to  fit  a  shadow  edge  in  one  orientation/ 
given  the  relative  Illumination  on  both  sides  of  a  line. 
However/  If  the  object  happened  to  have  a  darker  surface  than 
the  table/  this  heuristic  would  not  help. 

Clearly  this  procedure  leaves  many  unsolved  problems. 

In  general  there  will  bs  a  number  of  possible  labelings  from 
which  a  program  must  still  choose  one.  What  rules  can  It  use 
to  make  the  choice?  Even  after  choosing  a  labeling/  In  order 
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to  answer  questions  (about  the  number  of  objects  In  the 
scene/  about  which  edges  are  shadows/  about  whether  or  not 
any  objects  support  other  objects/  etc.)  a  program  must  use 
rules  of  some  sort  to  deduce  the  answers  from  the  Information 
It  has. 

I  will  argue  that  what  Is  needed  to  find  a  single 
reasonable  Interpretation  of  a  line  drawing  Is  not  a  more 
clever  set  of  rules  or  theorems  to  relate  various  features  of 
the  line  drawing/  but  merely  a  better  description  of  the 
scene  features.  In  fact/  it  turns  out  that  we  can  use  a 
parsing  procedure  which  involves  less  computation  than  the 
tree  search  procedure. 

2.3  BETTER  EDGE  DESCRIPTION 

So  far  I  have  classified  edges  only  on  the  basis  of 
geometry  (concave/  convex/  obscuring  or  planar)  and  have 
subdivided  the  planar  class  Into  crack  and  shadow 
sub-classes.  Suppose  that  I  further  break  down  each  class 
according  to  whether  or  not  each  edge  can  be  the  bounding 
edge  of  an  object.  Objects  can  be  bounded  by  obscuring 
edgeS/  concave  edgeS/  and  crack  edges.  Figure  2.5  shows  the 
results  of  appending  a  label  analogous  to  the  "obscuring 
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edge"  mark  to  crack  and  concave  edges.  This  approach  is 
similar  to  one  first  proposed  by  Freuder  (Freuder  1971a). 

Each  region  can  also  be  labeled  as  belonging  to  one  of 
the  three  following  classes: 

I  -  Illuminated  directly  by  the  light  source. 

SP  -  A  projected  shadow  region;  such  a  region  would  be 
illuminated  if  no  object  were  between  It  and  the  light 
source. 

SS  -  A  sel f -shadowed  region;  such  a  region  Is  oriented 
away  from  the  light  source. 

Given  these  classes^  1  can  define  new  edge  labels  which 
also  include  information  about  the  lighting  on  both  sides  of 
the  edge.  Notice  that  in  this  way  I  can  include  at  the  edge 
level/  a  very  local  level/  information  which  constrains  all 
edges  bounding  the  same  two  regions.  Put  another  way/ 
whenever  a  line  can  be  assigned  a  single  label  which  includes 
this  lighting  Information/  then  a  program  has  powerfui 
constraints  for  the  junctions  which  can  appear  around  either 
of  the  regions  which  bound  this  line. 
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Figure  2.6  Is  made  up  of  tables  which  relate  the  region 
illumination  types  which  can  occur  on  both  sides  of  each  edge 
type.  For  example#  If  either  side  of  a  concave  or  crack  edge 
Is  Illuminated#  both  sides  of  the  edge  must  be  Mluminated. 

These  tables  can  be  used  to  expand  the  set  of  allowable 
junction  labels;  the  new  set  of  labels  can  have  a  number  of 
entries  which  have  the  same  edge  geometries  hut  which  have 
different  region  Illumination  values.  It  Is  very  easy  to 
write  a  program  to  expand  the  set  of  labelings;  the 
principles  of  Its  operation  are  (1)  each  region  In  a  given 
junction  labeling  can  have  only  one  Illumination  value  of  the 
three#  and  (2)  the  values  on  either  side  of  each  line  of  the 
junction  must  satisfy  the  restrictions  In  the  tables  of 
figure  2.6. 

An  Interesting  result  of  this  further  subdivision  of  the 
line  labels  Is  that#  with  four  exceptions#  each 
shadow-causi ng  junction  has  only  one  possible  Illumination 
parsing#  as  shown  in  figure  2.7.  Thus  whenever  a  scene  has 
shadows  and  whenever  a  program  can  find  a  shadow  causing 
Junction  in  such  a  scene#  It  can  greatly  constrain  all  the 
lines  and  regions  which  make  up  this  junction.  In  figure  2.7 
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I  have  also  marked  each  shadow  edge  which  Is  part  of  a 
shadow-causing  junction  with  an  “L”  If  the  arrow  on  the 
shadow  edge  points  counter-clockwise  and  an  "R"  If  the  arrow 
points  clockwise.  No  "L"  shadow  edge  can  match  an  "R”  shadow 
edge^  corresponding  to  the  physical  fact  that  It  Is 
impossible  for  a  shadow  edge  to  be  caused  from  both  of  Its 
ends. 

There  are  two  extreme  possibilities  that  this 

partitioning  may  have  on  the  number  of  junction  labelings  now 

( 

needed  to  describe  all  real  vertices: 

(1)  Each  old  junction  label  which  has  n  concave  edges»  m 
crack  edges^,  p  clockwise  shadow  edges/  q  counterclockwise 
shadow  edges,  s  obscuring  edges  and  t  convex  edges  will  have 
to  be  replaced  by  (2o/*(6r'(3  )P(3)‘l(  9)^(8)^  new  junctions,  or 

(2)  Each  old  junction  will  give  rise  to  only  one  new 
junction  (as  In  the  shadow-causing  junction  cases). 

If  (1)  were  true  then  the  partition  would  be  worthless, 
since  no  new  information  could  be  gained.  If  (2)  were  true, 
the  situation  would  be  greatly  Improved,  since  In  a  sense  all 
the  much  more  precise  Information  was  implicitly  Included  In 
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the  orli‘na1  junctions  but  was  not  explicitly  stated. 

Because  the  Information  Is  now  more  explicitly  stated^  many 
matches  between  junctions  can  be  precluded;  for  example/  If 
In  the  old  scheme  some  line  segment  LI  of  junction  label  Q1 
could  have  been  labeled  concave/  as  could  line  segment  L2  of 
junction  label  Q2/  n  line  joining  these  two  junctions  could 
have  been  labeled  concave.  But  In  the  new  scheme/  If  each 
junction  label  gives  rise  to  a  single  new  label,  both  LI  and 
L2  would  take  on  one  of  the  twenty  possible  values  for  a 
concave  edge.  Unless  both  LI  and  L2  gave  rise  to  the  same 
new  label/  the  line  segment  could  not  be  labe,.J  concave 
using  Q1  and  Q2.  The  truth  lies  somewhere  between  the  two 
extremes/  but  the  fact  that  It  Is  not  at  the  extreme  of  (1) 
means  that  there  Is  a  net  Improvement.  In  Table  2.2  I 
compare  the  situation  now  to  cases  (1)  and  (2)  above  and  also 
to  the  situation  depicted  in  Table  2.1. 

I  have  also  used  the  better  descriptions  to  express  the 
restriction  that  each  scene  Is  assumed  to  be  on  a  horizontal 
table  which  has  no  holes  In  It  and  which  Is  large  enough  to 
fill  the  retina.  This  means  that  any  line  segment  which 
separates  the  background  (table)  from  the  rest  of  the  scene 
can  only  be  labeled  as  shown  In  figure  2.8.  Because  of  this 
fact  the  number  of  junction  labels  which  could  be  used  to 
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label  junctions  on  the  scene/background  boundary  can  be 
greatly  restricted. 

The  value  of  a  better  description  should  be  Immediately 
apparent.  In  the  old  classification  scheme  three  out  of  the 
seven  line  labels  could  appear  on  the  scene/background 
boundary#  whereas  In  the  new  classification#  only  seven  out 
of  fifty  labels  can  occur.  Moreover#  since  each  junction 
must  have  two  of  Its  line  segments  bounding  any  region#  the 
fraction  of  junctions  which  can  be  on  the  scene/background 
boundary  has  Improved  roughly  from  (3/7H3/7)  •  9/49  •  18. 4!^ 
to  (7/57)(7/57)  -  49/3149  «  1.6%.  The  results  of  these 
Improvements  will  become  obvious  In  the  next  section. 

2.4  PROGRAMMING  CONSEQUENCES 

There  are  so  many  possible  labels  for  each  type  of 
junction  that  I  decided  to  begin  programming  a  labeling 
system  by  writing  a  sort  of  filtering  program  to  eliminate  as 
many  junction  labels  as  possible  before  beginning  a  tree 
search  procedure. 


U 
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The  filter  procedure  depends  on  the  following 
observation/  given  In  terms  of  the  jigsaw  puzzle  analogy: 


Suppose  that  we  have  two  junctions#  J1  and  J2  which  are 
joined  by  a  line  segment  L-J1-J2.  J1  and  J2  are 
represented  by  adjacent  spaces  on  the  board  and  the 
possible  labels  for  each  junction  by  two  stacks  of 
pieces.  Now  for  any  piece  M  In  Jl’s  stack  either  (1) 
there  Is  a  matching  piece  N  In  d2's  stack  or  (2)  there 
Is  no  such  piece.  If  there  Is  no  matching  piece  for  M 
then  M  can  be  thrown  away  and  need  never  he  considered 
again  as  a  possible  junction  label. 


The  filter  procedure  below  Is  a  method  for 
systematically  eliminating  all  junction  labels  for  which 
there  can  never  be  a  match.  All  the  equipment  Is  the  same  as 
that  used  In  the  tree  search  example#  except  that  this  time  I 
have  added  a  card  marked  "junction  modified"  on  one  side  and 
"no  junction  modified"  on  the  other. 


Step  1:  Put  a  junction  number  tag  between  1  and  n  In 
each  "junction  number"  bln.  Place  a  full  set  of  pieces 
In  the  "untried  labels"  bln  of  each  junction. 

Step  2:  Set  the  counter  to  Nc  »  1#  and  place  the  card  so 
that  It  reads  "no  junction  modified". 

Step  3:  Check  the  value  of  Nc: 

A.  I f  Nc  «  n  ♦  1#  and  the  card  reads 'ho  junction 
modified"  then  go  to  SUCCEED. 

B.  If  Nc  =*  n  ♦  1#  and  the  card  reads  "junction 
modified"  then  go  to  Step  2.  (At  least  one  piece  was 
thrown  away  on  the  last  pass#  and  therefore  It  Is 
possible  that  other  pieces  which  were  kept  only  because 
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this  piece  was  present  will  now  have  to  be  thrown  av.ay 
a) so. } 

C.  Otherwise#  go  to  Step  4. 

Step  4:  Check  the  "untried  labels"  bln  of  junction 
J(Nc); 


A.  If  there  are  no  pieces  left  In  the  Nc-th 
"untried  labels"  bln#  then 

Al.  If  there  are  no  pieces  In  the  Nc-th 
"tried  labels"  bln#  go  to  FAILURE. 

A2.  Otherwise#  transfer  the  pieces  from  the 
Nc-th  "tried  labels"  bln  back  Into  the  Nc-th  "untried 
labels"  bln#  add  1  to  the  counter  (Nc)  and  go  to  Step  3. 

B.  If  there  are  pieces  left  In  the  Nc-th  "untried 
labels"  bln#  take  the  top  piece  from  the  bln  and  place 
It  In  the  board#  and  go  to  Step  5. 

Step  5:  Check  the  spaces  adjacent  to  space  Nc: 

A.  If  the  piece  In  the  Nc-th  space  has  matching 
pieces  In  each  neighboring  junction  space#  transfer  the 
piece  from  space  Nc  Into  the  Nc-th  "tried  labels"  bln# 
and  transfer  the  pieces  from  the  neighboring  spaces  and 
the  neighboring  "tried  labels"  bins  back  Into  their 
"untried  labels"  bins. 

B.  If  there  are  empty  neighboring  spaces#  then 

BX.  If  there  are  no  more  junctions  In  the 
neighboring  "untried  labels"  bins  which  could  fit  with 
the  piece  In  space  Nc#  then  that  piece  Is  not  a  possible 
label.  Throw  It  away#  and  arrange  the  card  to  read 
"junction  modified"  If  It  doesn't  already. 

B2.  Try  pieces  from  the  neighboring  "untried 
labels"  piles  until  either  a  piece  fits  or  the  pile  Is 
exhausted#  and  then  go  to  Step  5  again. 

SUCCEED:  The  pieces  In  the  "untried  labels"  bins  of  each 
junction  have  passed  the  filtering  routine  and 
constitute  the  output  of  this  procedure. 
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FAILURE:  There  Is  no  way  to  label  the  scene  given  the 

current  set  of  pieces. 

In  the  program  I  wrote^  I  used  a  somewhat  more  complex 
variation  of  this  procedure  which  only  requires  oie  pass 
through  the  junctions.  This  procedure  Is  similar  to  the  one 
used  to  generate  figure  2.4/  and  Is  described  below. 

When  I  ran  the  filter  program  on  some  simple  line 
drawings/  I  found  to  my  amazement  that  the  filter  procedure 
yielded  unique  labels  for  each  junction  In  most  cases!  In 
fact  In  every  case  I  have  tr^ed/  the  results  of  this 
filtering  program  are  the  same  results  which  would  be 
obtained  by  running  a  tree  search  procedure/  saving  all  the 
labelings  produced,  and  combining  all  the  resulting 
possibilities  for  each  junction.  In  other  wordS/  the  filter 
program  In  general  eliminates  all  labels  except  those  which 
are  part  of  some  tree  search  labeling  for  the  entire  scene. 

It  Is  not  obvious  that  this  should  be  the  case.  For 
example/  If  this  filter  procedure  Is  applied  to  the  simple 
line  drawing  shown  In  figure  2.4  using  the  old  set  of  labels 
given  In  figure  2.3/  It  produces  the  results  shown  In  figure 
2.9.  in  this  figure/  each  junction  has  labels  attached  which 
would  not  be  part  of  any  total  labeling  produced  by  a  tree 
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search.  This  figure  is  obtained  by  going  through  the 
junctions  In  numerical  order  and: 

(1)  Attaching  to  a  junction  all  labels  which  do  not 
conflict  with  junctions  previously  assigned;  !,e.  If  It  Is 
known  that  a  branch  must  be  labeled  from  the  set  S#  do  not 
attach  any  junction  labels  which  would  require  that  the 
branch  be  labeled  with  an  element  not  in  S. 

(2)  Looking  at  the  neighbors  of  this  junction  which  have 
already  been  labeled;  If  any  label  does  not  have  a 
corresponding  assignment  for  the  same  branch#  then  eliminate 

i  t. 


(3)  Whenever  any  label  is  deleted  from  a  junction#  look 
at  all  its  neighbors  In  turn#  and  see  if  any  of  their  labels 
can  be  eliminated.  If  they  can#  continue  this  process 
Iteratively  until  no  more  changes  can  be  made.  Then  go  on  to 
the  next  junction  (numerically).  The  junction  which  was 
being  labeled  (as  in  step  (D)  at  the  time  a  label  was 
eliminated  (struck  out  In  the  figure)  is  noted  next  to  each 
eliminated  label  in  figure  2.9. 
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The  fact  that  these  results  can  be  produced  by  the 
filtering  program  says  a  great  deal  about  line  drawings 
generated  by  real  scenes  and  also  about  the  value  of  precise 
descriptions.  There  Is  sufficient  local  Information  In  a 
line  drawing  so  that  a  program  can  use  a  procedure  which 
requires  far  less  computation  than  does  a  tree  search 
procedure.  To  sec  why  this  Is  sc#  notice  that  If  the 
description  the  program  uses  Is  good  enough#  then  many 
junctions  must  always  be  given  the  same  unique  label  in  each 
tree  search  solution;  the  filtering  program  needs  to  find 
such  a  label  only  once#  while  a  tree  search  procedure  must  go 
through  the  process  of  finding  the  same  solution  on  each  pass 
through  the  tree. 

Quite  remarkably#  all  these  results  are  obtained  u-:ing 
only  the  topology  of  line  drawings  plus  knowledge  about  which 
region  is  the  table  and  about  the  relative  brightness  of  each 
region.  No  use  is  made  (yet)  of  the  direction  of  line 
segments  (except  that  some  directional  information  Is  used  to 
classify  the  junctions  as  ARROWs#  FORKs#  etc.)#  nor  is  any 
use  made  of  the  length  of  line  segments#  microstructure 
edges#  lighting  direction  or  other  potentially  useful  cues. 
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2,5  HANDLING  BAD  DATA 

So  far  5  have  treated  this  subject  as  though  the  program 
would  always  be  given  perfect  data.  In  fact  there  are  many 
types  of  errors  and  degeneracies  which  occur  frequently. 

Some  of  these  can  be  corrected  through  use  of  better  line 
finding  programs  and  some  can  be  eliminated  by  using  stereo 
information^  but  i  would  like  to  show  that  the  program  can 
handle  various  problems  by  simple  extensions  of  "he  list  of 
Junction  labels.  In  no  case  do  I  expect  the  program  to  be 
able  to  sort  out  scenes  that  people  cannot  easily  understand. 

Two  of  the  most  common  types  of  bad  data  are  (1)  edges 
missed  entirely  due  to  equal  region  brightness  on  both  sides 
of  the  edge/  and  (2)  accidental  alignment  of  vertices  and 
lines.  Figure  2.10  shows  a  scene  containing  Instances  of 
each  type  of  problem. 

The  program  handles  these  problem  junctions  by 
generating  labels  for  them,  just  as  It  does  for  normal 
junctions.  It  is  important  to  be  able  to  do  thiS/  since  it 
Is  In  general  very  difficult  to  identify  the  particular 
junction  which  causes  the  program  to  fail  to  find  a  parsing 
of  the  scene*  Even  worse/  the  program  may  find  a  way  of 
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Interpreting  the  scene  as  though  the  data  were  perfect  and  It 
would  then  not  even  get  an  indication  that  it  should  look  for 
other  Interpretations. 

2.6  ACCIDENTAL  ALIGM1ENT 

Chapter  7  treats  a  number  of  different  types  of 
accidental  alignment.  Figure  2.11  shows  three  of  the  most 
common  types  which  are  Included  in  the  program's  repertoire; 
consider  three  kinds  of  accidental  alignment: 

(1)  cases  where  a  vertex  apparently  has  an  extra  line 
because  an  edge  obscured  by  the  vertex  appears  to  be  part  of 
the  vertex  (see  figure  2.11a}, 

(2)  cases  where  an  edge  which  Is  between  the  eye  and  a 
vertex  appears  to  intersect  the  vertex  (see  figure  2.  lib), » 
and 

(3)  cases  where  a  shadow  is  projected  so  that  it 

o 

actually  does  intersect  a  vertex  (.,see  figure  2.11c). 
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2.7  MISSING  LINES 

I  have  not  attempted  to  systematically  Include  all 
missing  line  possibilities^  but  have  only  Included  labels  for 
the  most  corranon  types  of  missing  lines.  I  require  that  any 
missing  line  be  In  the  Interior  of  the  scene;  no  line  on  the 
scene/background  boundary  can  be  missing.  I  also  assume  that 
all  objects  have  approximately  the  same  reflectivity  on  all 
surfaces.  Therefore,  If  a  convex  line  Is  missing,  I  assume 
that  either  both  sides  of  the-  edge  were  Illuminated  or  that 
both  were  shadowed.  I  have  not  really  treated  missing  lines 
In  a  complete  enough  way  to  say  much  about  them.  There  vl  II 
have  to  be  facilities  In  the  program  for  filling  In  hidden 
surfaces  and  back  faces  of  objects  before  missing  lines  can 
be  treated  satisfactorily. 

In  general  the  program  will  report  that  It  Isunable  to 
label  a  scene  If  more  than  a  few  lines  are  missing  and  the 
missing  line  labels  aro  not  Included  In  the  set  of  possible 
junction  labels.  This  Is  really  a  sign  of  the  power  of  the 
program,  since  If  the  appropriate  labels  for  the  missing  line 
junctions  were  Incl  Ked,  the  program  would  find  them 
uniquely.  As  an  example,  the  simple  scene  In  figure  2.12 
cannot  be  labeled  at  all  unless  the  missing  line  junctions 
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are  included. 


2.8  REGION  ORIENTATIONS 


Regions  can  be  assigned  labels  which  give  quantized 
values  for  region  orientations  in  three  dimensions.  These 
labels  can  be  added  to  the  junction  labels  in  very  much  the 
sanie  way  that  the  region  illumination  values  were  added.  It 
is  impossible  to  do  justice  to  the  topic  here^  but  region 
orientations  are  treated  in  considerable  detail  in  Chapter  8. 
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3.0  TRIHEDRAL  JUNCTION  LABELS 

The  knowledge  of  this  system  is  expressed  In  several 
distinct  forms: 

(1)  A  list  of  possible  junction  labels  for  each  type  of 
junction  geometry  includes  the  a  priori  knowledge  about  the 
possible  three  dimensional  interpretations  of  a  junction. 

(2)  Selection  rules  which  use  junction  geometry, 
knowledge  about  which  region  is  the  table,  anc  region 
brightness.  These  can  easily  be  extended  to  use  line  segment 
directions  to  find  the  subset  of  the  total  list  of  possible 
junction  labelings  which  could  apply  at  a  particular  junction 
in  a  line  drawing. 

(3)  A  program  to  find  the  possible  labelings;  It  knows 
how  to  systematically  eliminate  Impossible  combinations  of 
labels  in  a  line  drawing  and,  as  such,  contains  implicit 
knowledge  about  topology. 

(4)  Optional  heuristics  which  can  be  invoked  to  select 
a  single  labeling  from  among  those  which  remain  after  all  the 
other  knowledge  in  the  program  has  been  used.  These 
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heuristics  find  a  "plausible**  Interpretation  If  required. 

For  example^  one  heuristic  eliminates  Interpretations  that 
Involve  concave  objects  In  favor  of  ones  that  Involve  convex 
objects/  and  another  prefers  Interpretations  which  have  the 
smallest  number  of  objects;  this  heuristic  prefers  a  shadow 
Interpretation  for  an  ambiguous  region  to  the  Interpretation 
of  the  region  as  a  piece  of  an  object. 

In  this  chapter  I  show  how  to  express  the  first  type  of 
knowledge/  and  give  hints  about  some  of  the  others.  A  large 
proportion  of  my  energy  and  thought  has  gone  Into  the  choice 
of  the  set  of  possible  line  labels  and  the  sets  of  possible 
junction  labels.  In  this  I  have  been  guided  by  experiment 
with  my  program/  since  there  are  simply  too  many  labels  to 
hand  simulate  the  program's  reaction  to  a  scene.  The 
program/  the  set  of  edge  labels/  and  the  sets  of  junction 
labelings  have  each  gone  through  an  evolution  Involving 
several  steps.  At  each  step  I  noted  the  ambiguities  of 
interpretation  which  remained/  and  then  modified  the  system 
appropriately. 

The  changes  have  generally  Involved  (1)  the  subdivision 
of  one  or  more  edge  labels  Into  several  new  labels  embodying 
finer  distinctions/  and  (2)  the  recomputation  of  the  junction 
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label  lists  to  include  these  new  distinctions.  In  each  case 
I  have  been  able  to  test  the  new  scheme  to  make  sure  that  It 
solves  the  old  problems  without  creating  any  unexpected  new 
ones.  For  example#  the  Initial  data  base  contained  only 
junctions  which  (1)  represented  trihedral  vertices  (I.e, 
vertices  caused  by  the  Intersection  of  exactly  three  planes 
at  a  point  In  space)  and  (2)  which  could  be  constructed  using 
only  convex  objects.  The  present  data  base  has  been  expanded 
to  include  all  trihedral  junctions  and  a  number  of  other 
junctions  caused  by  vertices  where  more  than  three  planes 
meet. 

Throughout  this  evolutionary  process  I  have  tried  to 
systematically  include  in  the  lists  every  possibility  under 
the  stated  assumptions.  In  this  part  of  the  system  I  have 
made  only  one  type  of  judgement:  If  a  junction  can  represent 
a  vertex  which  is  physically  possible#  Include  that  junction 
in  the  data  base. 

3.1  EDGE  GEOMETRY 

Jf 

The  first  problem  Is  to  find  all  possible  trihedral 
vertices.  Huffman  observed  that  three  Intersecting  planes# 
whether  mutually  orthogonal  or  not#  divide  space  into  eight 
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parts  so  that  the  types  of  trihedral  vertex  can  be 
characterized  by  the  octants  of  space  around  the  vertex  which 
are  filled  by  solid  material  (Huffman  1971). 

Dowson  (Dowson  1971a)  went  a  little  further  In 
discussing  how  one  could  write  an  algorithm  to  find  all 
possible  trihedral  Junctions  and  their  labels  (using  the 
simple  three-label  model  of  Huffman  and  Clowes).  In  fact  he 
never  used  his  system  to  generate  every  class  of  junction 
geometry  but  was  satisfied  to  show  that  it  could  generate  the 
twelve  labels  which  Huffman  and  Clowes  originally  used. 

These  twelve  labels  represent  four  different  ways  of  filling 
In  the  octants  (where  I  have  not  counted  ways  of  filling  the 
octants  which  differ  only  by  rotation  as  different). 

Dowson 's  scheme  Is  useful  for  visualizing  how  to 
generate  the  ten  different  ways  of  filling  the  octants  which 
I  use.  Consider  the  general  Intersection  of  three  planes  as 
shown  In  figure  3.1.  These  planes  divide  space  Into  octants^ 
which  can  be  uniquely  identified  by  three-dimensional  binary 
vectors  (x  y  ^)  where  the  x^  y,  and  z  directions  are 
specified  as  shown.  The  vectors  make  It  easy  to  describe  the 
various  geometries  precisely.  I  can  then  generate  all 
possible  geometries  and  non-degenerate  views  by  Imagining 
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FIGURE  3.1 
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various  octants  to  be  filled  in  with  solid  material.  There 
are  junctions  which  correspond  to  having  1,  2,  3#  U,  5,  6,  or 
7  octants  filled.  Figure  3.2  shows  the  twenty  possible 
geometries  that  result  from  filling  various  octants^  and  In 
Appendix  1  I  have  shown  all  the  Junction  labelings  (not 
including  shadow  variations)  which  can  result  from  the 
geometries  in  figure  3.2A.  The  result  of  this  process  is  196 
different  junction  labels.  Figure  3.2B  consists  of  the 
geometries  which  I  have  chosen  not  to  us'  to  generate 
junction  labels.  I  have  not  included  these  geometries  because 
each  involves  objects  which  touch  only  along  one  edge^  and 
whose  faces  are  nonetheless  aligned^  an  extremely  unlikely 
arrangement  when  compared  to  the  other  geometries.  (In 
addition#  some  of  the  geometries  are  physically  Impossible 
unless  one  or  more  objects  are  cemented  together  along  an 
edge  or  supported  by  invisible  means.) 

The  four  geometries  recognized  by  Huffman#  Clowes#  and 
Dowson  correspond  to  my  numbers  1#  3#  5#  and  7  in  figure 
3.2A. 

In  figure  3.3  I  show  how  the  20  different  Irbels  with 
type  3  geometry  can  be  generated.  Basically  this  process 
involves  taking  a  geometry  from  figure  3.2A#  finding  all  the 
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ways  that  the  solid  segments  can  be  connected  or  separated^ 
and  finding  all  the  possible  views  for  each  partitioning  of 
the  quadrants.  To  generate  all  the  possible  views  one  can 
either  draw  or  Imagine  the  particular  geometry  as  It  appears 
when  viewed  from  each  octant.  From  some  viewing  octants  the 
central  vertex  Is  blocked  from  view  by  solid  material#  and 
therefore  not  every  viewing  position  adds  new  labelings. 
Appendix  1  Is  obtained  by  applying  this  process  to  each  of 
the  geometries  In  figure  3.2A. 

Whenever  one  of  the  regions  at  a  junction  could 
correspond  to  the  background  (l.e.  the  region  Is  not  part  of 
one  of  the  three  planes  which  Intersect  at  the  vertex)  I  have 
marked  the  region  with  a  star  (★)  both  In  figure  3.3  and 
Appendix  1.  Later  I  will  show  how  to  use  this  Information  to 
aid  the  selection  rules.  Only  37  ojt  of  the  196  labels  In 
Appendix  1  can  occur  on  the  scene/backgrou'’d  boundary. 

3.2  A  USEFUL  HEURISTIC 

This  section  previews  the  general  discussion  later 
concerning  how  to  choose  a  single  labeling  If  ambiguities  are 
still  left  at  the  end  of  the  regular  program's  operation. 

The  regular  program  keeps  every  conceivable  Interpretation, 
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Clearly  in  some  cases  the  scene  is  essentially  ambiguous^ 
i.e.  human  beings  can  interpret  the  scene  in  more  than  one 
way. 

Given  the  line  drawing  shown  in  figure  3.4,  how  can  a 
program  decide  which  of  the  Interpretations,  A,  B,  C  or  D,  is 
"correct"?  In  a  picture  there  may  be  cues  about  how  the 
objects  sr,ou1d  be  separated  in  the  details  of  the  edges 
L-J1-J2  and  L-J2-J3  of  figure  3,4.  But  given  only  the  line 
drawing  of  figure  3.4,  the  program  will  find  the  four 
interpretations  listed.  Because  we  generally  prefer  the 
scene  Interpretation  which  has  the  smallest  number  of  convex 
objects,  I  have  appropri atel y  marked  all  junction  labelings 
which  include  either  concave  edges  (whether  visible  or  not) 
or  three-object  edges.  The  output  of  the  regular  program  Is 
then  a  single  label  or  list  of  labels  for  each  junction. 
Obviously  If  there  Is  only  a  single  label,  then  there  Is 
nothing  left  to  do.  But  If  more  than  one  label  Is  left,  it 
can  purge  labels  corresponding  to  concave  or  three-edge 
junctions. 

This  heuristic  correctly  labels  all  the  scenes  shown  . 
fi:.ure  3 .5 A,  but  finds  the  wrong  labeling  for  figure  3.5B 
because  It  always  prefers  to  Interpret  scenes  as  made  up  of 
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convex  objects/  and  does  not  know  enough  to  preclude  the 
convex  labeling  In  this  case  because  object  A  In  figure  3. SB 
has  no  support.  Of  course/  for  ambiguous  scenes  like  figure 
3.4  the  ‘^I'uristlc  selects  Interpretation  A. 

3.3  SHADOWS  AT  TRIHEDRAL  VERTICES 

To  find  all  the  variations  of  these  verti  ces  vhi  ch 
Include  shadow  edgeS/  first  note  that  vertices  with  1/  2/  B 
or  7  octants  filled  cannot  cause  shadows  such  lhat  the  shadow 
edges  appear  as  part  of  the  vertex.  This  can  be  statec".  more 
generally:  in  order  to  be  a  shadow-causing  vertex  (I.e.  a 
vertex  where  the  caused  shadow  edge  radiates  from  the  vertex) 
there  must  exist  some  viewing  position  for  the  vertex  from 
which  either  two  concave  edges  and  one  convex  edge  or  one 
concave  edge  and  two  convex  edges  are  visible.  Consider  the 
geometries  listed  in  figure  3.2A.  First/  a  shadow-causing 
edge  must  be  convex.  Second/  unless  there  Is  at  least  one 
concave  edge  adjacent  to  this  convex  edge/  there  can  be  no 
surface  which  can  have  a  shadow  projected  onto  It  by  the 
light  streaming  by  the  convex  edge.  Finally/  a  junction 
which  has  one  convex  and  one  concave  edge  must  have  at  least 
one  other  convex  or  concave  edge/  since  the  convex  edge  and 
concave  edge  define  at  least  three  planes  which  cannot  meet 
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at  any  vertex  with  only  two  edges. 

This  Immediately  eliminates  73  out  of  196  cf  the  labels 
In  Appendix  1  from  consideration.  Appendix  2  shows  the 
shadow  edges  (’f  any)  which  can  occur  at  each  of  the 
remaining  vertices.  Appendix  2  Is  constructed  In  the  manner 
Illustrated  In  figure  3.6;  for  each  potential  shadow-causing 
vertex,  imagine  the  light  source  to  be  In  each  of  the  octants 
surrounding  the  vertex,  and  record  all  the  resulting 
junctions.  1  have  marked  each  shadow  edge  which  is  part  of  a 
shadow-causing  junction  with  an  "L"  or  "R"  according  to 
whether  the  arrow  on  the  shadow  edge  points  counterclockwise 
or  clockv/lse  respectively. 

Any  junction  which  contains  either  a  clockwise  shadow 
edge,  marked  "R,"  or  a  counterclockwise  shadow  edge,  marked 
"L,"  Is  defined  as  a  shadow-causing  junction.  The  reason  for 
distinguishing  between  the  L  and  R  shadow  edges  Is  that  this 
prevents  labeling  an  edge  as  if  It  were  a  shadow  caused  from 
both  its  vertices.  Without  this  device  there  would  be  no  way 
to  prevent  figure  3.7  from  being  labeled  as  shown,  with  line 
segment  L-A-B  interpreted  as  a  shadow  edge.  (I  use  "L-"  as  a 
prefix  to  mean  “line  segment(s)  joining  the  following 
points";  thus  L-A-B  Is  the  line  segment  joining  points  A  and 
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B.)  When  the  "L"  and  "R"  marks  are  attached  to  ea  .h  shadow 
causing  Junction/  then  the  two  shadow  causing  junctions  at  A 
and  B  In  figure  3.7  no  longer  are  compatible/  and  therefore 
the  labeling  shown  will  not  be  considered  possible  by  the 
program. 

3.4  OTHER  NON-DEGENERATE  JUNCTIONS 

I  now  must  describe  vertices  which  do  not  fall  Into  the 
categories  I  have  described  so  far.  These  Include  (1)  all 
the  rest  of  the  combinations  that  shadow  edges  can  form  and 
(2)  obscured  edges. 

In  figure  3.8A  I  show  all  the  other  non-degenerate 
vertices  which  Involve  shadow  edgeS/  and  In  figure  3.8B  I 
show  al 1  the  obscured  edges. 

Later  I  return  to  the  topic  of  junction  labels  and  show 
how  It  Is  possible  to  also  Include  junctions  representing 
common  degeneracies  and  accidental  alignments  as  well  as 
junctions  with  missing  lines.  In  the  degenerate  cases  I  do 
not  Include  every  labeling  possibility;  Instead  I  Include 
the  most  common  occurrences  using  certain  observations  about 
junctions.  This  Is  Important  since  I  do  not  want  to  limit  the 
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program  to  any  particular  set  of  objects.  Fortunately 
certain  types  of  junctions  are  rare  no  matter  what  types  of 
objects  are  in  a  scene;  for  example^  many  junctions  can  only 
occur  when  the  eye^  tight  and  object  are  aligned  to  within  a 
few  degrees^  and  when  these  junctions  also  contain  unusual  or 
aligned  edges  the  combined  likelihood  of  the  junctions  is  low 
enough  so  that  they  can  be  safely  omitted.  As  shown  in 
Chapter  7,  the  program  can  still  give  information  about 
junctions  even  If  they  do  not  have  proper  labelings  listed  in 
the  data  base^  provided  that  not  too  many  of  these  occur 
together  in  a  single  scene.  Moreover,  this  approach  is 
reasonable,  since  any  additional  ability  to  use  stereo  images 
or  to  move  the  eye  or  range-finding  ability  will  allow  a 
program  to  disambiguate  most  of  these  types  of  features. 


3.5  A  CLASS  OF  DEGENERACIES 


As  a  final  topic,  I  include  one  type  of  degeneracy  which 
cannot  be  resolved  by  eye  motion  or  stereo.  This  type  of 
degeneracy  results  when  the  light  source  is  placed  in  the 
plane  defined  by  one  of  an  object's  faces.  In  this  case, 
shadows  are  aligned  with  edges  to  produce  junctions  which  are 
unlabelable  given  only  the  normal  set  of  labels  described  so 
far.  Two  examples  of  such  alignment  are  shown  in  figure  3.9A 


and  figure  3.9B  and  a  complete  listing  of  this  type  of 
Junction  is  found  in  Appendix  3.  I  have  excluded  cases  where 
a  shadow  edge  is  projected  directly  onto  an  edge  of  some 
other  type  (as  in  figure  3.90.  These  cases  are  excluded 
since  they  would  require  me  to  define  new  edge  labels  which 
are  of  very  limited  value/  although  there  Is  no  technical 
difficulty  in  defining  such  edges  and  junctions.  I  also  have 
excluded/  for  the  time  being/  cases  like  the  one  shown  in 
figure  3.90/  since  the  two  junctions  marked  only  appear  to  be 
T  junctions  when  the  eye  is  in  the  plane  defined  by  the  light 
source  and  the  shadow-causing  edge  (L-A-B  or  L-C-D  in  figure 
3.90).  If  the  eye  is  moved  to  the  right/  the  shadow-causing 
junctions  change  to  ARROWS  or  FORKS  as  illustrated  In  figure 
3.9E.  In  contrast/  notice  that  for  the  scenes  shown  in 
figures  3.9A  and  3.9B/  no  change  in  eye  position  can  make  any 
difference  in  the  apparent  geometry  of  the  shadow-causing 
junctions. 

Later  (in  Chapter  6)  I  consider  some  of  the  common 
non-tri hedral  junctions  which  the  program  is  likely  to 
encounter.  Some  of  these  require  me  to  define  extra  labels. 
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The  grand  total  number  of  legal  trihedral  junctions 
listed  In  this  chapter  Is  505.  The  Interesting  thing  In  my 
estimation  is  that  the  number  of  junction  labels^  while 
fairly  large^  Is  very  small  compared  to  the  number  of 
possibilities  if  the  branches  of  these  junctions  were  labeled 
independently;  moreover#  even  though  I  have  not  yet  shown 
how  to  include  various  degeneracies  and  alignments#  I  believe 
that  the  set  I  have  described  already  is  sufficient  for  most 
scenes  which  a  person  would  construct  out  of  p1ane~faced 
objects#  provided  that  he  did  not  set  out  to  deliberately 
confuse  the  program. 

Since  it  may  not  be  obvious  what  types  of  common 
vertices  are  non-tr 1 hedral #  figure  3.10  contains  a  number  of 
such  vertices.  Later  sections  show  how  to  handle  all  of 


them 


Figure  5.10 
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4.0  COMPLETING  THE  REGULAR  DATA  BASE 

It  would  be  hard  to  devise  a  program  which  could  start 
with  a  few  pieces  of  Information  and  eventually  yield  the 
list  of  junctions  described  In  Chapter  2.  Moreover,  even  If 
such  a  program  were  written  (which  would  Indeed  be 
theoretically  Interesting),  It  would  be  rather  pointless  to 
generate  labels  with  It  every  time  the  labels  are  needed  In 
an  analysis.  Instead  the  generating  program  could  run  once 
and  save  Its  results  In  a  table.  In  this  form  the  junction 
labelings  table  Is  a  sort  of  compiled  knowledge,  computed 
once  using  a  few  general  facts  and  methods.  The  knowledge  li 
the  current  program  Is  almost  totally  In  this  compiled  form; 
this  Is  the  reason  for  Its  rapid  operation,  but  I  have  paid  a 
price  for  this  speed  In  that  I  require  a  large  amount  of 
memory  (about  14,000  words)  to  store  the  junction  labelings. 
(All  the  rest  of  the  labeling  program  occupies  only  about 
4000  words  of  memory  even  though  It  Is  written  In 
MICRO-PLANNER  and  LISP,  neither  of  which  are  particularly 
noted  for  space  efficiency.) 
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4.1  REGION  ILLUMINATION  ASSIGNMENTS 

Given  tables  of  allowable  region  Illumination  values 
(figure  1.6),  It  Is  easy  to  show  how  to  write  a  program  which 
expands  the  data  base  to  Include  this  Information.  Suppose 
that  I  wish  to  expand  the  labeling  of  the  junction  shown  In 
figure  4.1  to  Include  region  Illumination  values.  As  coded 
for  the  data  base^  this  labeling  Is: 

(OCRM  PLUS  OCLM  SHCCL) 

where  OCRM  stands  for  OCcIude  Right  Minus  (see  L-J-A  in 
figure  4.1)/  PLUS  represents  the  convex  edge  (see  L'd-B  In 
figure  4.1)/  OCLM  stands  for  OCcIude  Left  Minus  (see  L-J-C  in 
figure  4.1)/  and  SHCCL  stands  for  SHadow  Counterclockwise 
type  L  (see  L-J-0  in  figure  4.1). 

Each  of  these  edges  can  separate  regions  which  have  the 
following  values  (the  first  element  is  the  value  of  the 
region  located  counterclockwise  with  respect  to  the  edge/  the 
second  element  Is  the  value  of  the  region  located  clockwise 
with  respect  to  the  edge): 
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<The  list  of  region  Mlumination  pairs  for  OCRM  or  0CLM> 

•  LI 

>  ((I  I)  (SP  SP)  (SP  SS)  (SS  SP)  (SS  SS)). 

<The  list  of  region  1 11 umi nation  pairs  for  PLUS> 

•  L2 

«  ((I  1)  (1  SS)  (SS  I) 

(SP  SP)  (SP  SS)  (SS  SP)  (SS  SS)). 

<rhe  list  of  region  illumination  pairs  for  SHCCL> 

«  L3 

•  ((SP  D). 

ILLUMINE  Is  a  function  which  takes  two  input  lists  as 
arguments/  and  returns  a  single  output  list.  Each  member  of 
the  output  list  Is  formed  as  follows:  take  a  member  of  the 
second  input  list  whose  first  element  Is  the  same  as  the  last 
element  of  some  member  of  the  first  Input  list.  Concatenate 
tliese  two  and  eliminate  the  duplication  of  the  matching 
element.  The  oupuv  list  Is  made  up  of  every  possible  element 
which  can  be  formed  In  this  manner.  While  a  verbal 
description  may  be  somewhat  difficult  to  understand/  the 
function  is  not  really  very  complicated/  and  I  think  the 
following  example  should  make  its  operation  clear.  Using  the 


SECTION  4.1  111 


lists  LI  and  L2  that  I  defined  earlier: 


ILLUMINE  (Ll«  L2) 


>  ((I  ID  (II  SS) 

(SP  SP  SP)  (SP  SP  SS) 

(SP  SS  I)  (SP  SS  SP)  (SP  SS  SS) 
(SS  SP  SP)  (SS  SP  SS) 

(SS  SS  I)  (SS  SS  SP)  (SS  SS  SS)) 

»  L4. 


L4  is  a  list  of  triples  which  gives  all  the  possible 
values  for  region  illuminations  in  the  regions  RO/  Rl/  and  R2 
in  figure  4.1.  To  include  R3/  compute  L5: 


ILLUMINE  (L4,  LI) 

-  ((I  I  I  I) 

(I  I  SS  SP)  (I  I  SS  SS) 

(SP  SP  SP  SP)  (SP  SP  SP  SS) 
(SP  SP  SS  SP)  (SP  SP  SS  SS) 
(SP  SS  I  I ) 

(SP  SS  SP  SP)  (SP  SS  SP  SS) 

(SP  SS  SS  SP)  (SP  SS  SS  SS) 

(SS  SP  SP  SP)  (SS  SP  SP  SS) 

(SS  SP  SS  SP)  (SS  SP  SS  SS) 

(SS  SS  I  I) 

(SS  SS  SP  SP)  (SS  SS  SP  SS) 

(SS  SS  SS  SP)  (SS  SS  SS  SS)) 

-  L5. 


Now  I  only  need  to  include  the  oairs  for  the  line  L-J-O/ 
the  shadow  edge.  Notice  that  very  few  of  the  possibilities 
for  illumination  can  agree  with  R3  when  R3  is  forced  to  be  a 
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type  SP  region: 

ILLUMINE  (L5/  L3} 


((I  1 

SS 

;  SP  I) 

(SP 

SP 

SP 

SP 

1 ) 

(SP 

SP 

SS 

SP 

1 ) 

(SP 

SS 

SP 

SP 

1 ) 

(SP 

SS 

SS 

SP 

1) 

(SS 

SP 

SP 

SP 

1 ) 

(SS 

SP 

SS 

SP 

1) 

(SS 

SS 

SP 

SP 

1 } 

(SS 

SS 

SS 

SP 

1 )) 

-  L6. 

Now/  to  find  the  labelings  for  this  junction#  the  last 
condition  requires  that  since  the  first  and  last  elements  of 
each  labeling  In  L6  both  refer  to  RO#  their  values  must  be 
the  same.  Therefore  I  apply  function  FINALIZE#  which  only 
keeps  members  of  a  list  whose  first  and  last  elements  are  the 
same: 


FINALIZE  (L6}  «  ((I  I  5S  SP  l)>. 

This  represents  the  only  possible  region  Illumination 
labeling  for  this  junction  as  shown  In  figure  4.2A.  As  I 
mentioned  earlier#  It  Is  true  In  general  that  shadow-causing 
junctions  (and  a  number  of  other  junctions  Involving  shadows) 
have  only  one  possible  region  illumination  labeling.  The 


exceptions  to  this  rule  are  shadow-causing  junctions  where 
one  region  segment  of  the  junction  is  obscured  by  the  vertex 
which  gives  rise  to  the  junction.  To  understand  this 
distinction^  try  finding  the  region  illumination  values  for 
the  junction  in  figure  4.2B  as  an  exercise,  especially  if  you 
are  not  entirely  clear  about  the  operation  cf  ILLUMINE  and 
FINALIZE.  You  will  need  the  list  of  possible  region 
illumination  pairs  for  L-V-A  and  L-V-D  in  figure  4.2B;  these 
edges  can  each  be  assigned  any  of  the  possible  regton 
i llumi nation  pairs: 

<The  list  of  region  i  1  lur.i  nation  pairs  for  OCR  edges  (such  as 
L-V-A)  and  OCL  edges  (such  as  L-V-D )> 

-  C(l  I)  (I  SP)  (I  SS) 

(SP  I )  (SP  SP)  (SP  SS) 

(SS  I)  (SS  SP)  (SS  SS)) 

-  L7. 

Your  answer  should  be: 


i 


I 


((I  SS  SP  I  I) 

(SP  SS  SP  I  SP) 
(SS  SS  SP  I  SS)) 


The  answer  is  illustrated  in  figure  4.2C. 


I 


I 

‘i 

j 


a 
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In  order  to  include  Illumination  Information  in  the  data 
base,  I  merely  append  the  region  illumination  value  names  to 
the  name  of  each  label.  Thus  I  subdivide  each  label  type 
(except  shadow  edge  labels)  into  a  number  of  possibilities, 
as  shown  in  Table  4,1.  As  I  mentioned  in  Chapter  2, 
expanding  the  number  of  line  labels  does  not  increase  the 
total  number  of  jiinction  labels  as  mu  ;h  as  one  might  imagine 
(see  Table  2.2). 

Fully  268  of  the  505  labelings  listed  in  Chapter  3,  over 
half,  have  only  one  possible  region  illumination 
Interpretation!  The  largest  possible  number  of  illumination 
Interpretations  for  any  junction  Is  5*^,  where  n  is  the  number 
of  junction  branches,  A  number  of  T  junctions  actually  have 
27  Interpretations  (for  example,  this  is  true  of  any  T  made 
up  of  three  occluding  edges). 
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NAMES  OF 
OU)  UVBEIS 

AfPEASANOE 

NEW  LABEIS 

$HCC 

CSKKl>CiW" 
c?o-mrEFi-  , 
ci*ockwisb) 

SHCC-SH 

SHCCL 

^  /j>l. 

S'HCCL-SPI 

SKCCK, 

_4S_ 

SHCCR-SPI 

shcl 

Wfc 

SHCITSP 

S'HClt. 

SHCtt-'lS? 

^HCUL 

•  ipL*" 

S/HCUl-ISP 

OCBK 

roCCUtCI»E  ^  . 

ocm'-ix,  OCRtvi-^gP, 
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ViuTV'AlHXi:^ 

cctn-xx,  ocjLisv-prsfP, 
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OCL?<\"S^$$ 

(dOWcJAYB 

mOINEX>P 

JVI-II 

iA-2SSP> 

.nx 

cco^cc^T.a  - , 

-S  OBJECTS) 

74X-XI, 

7AX*S5^Ps 

OCK.CR. 

OCRCR-'II,  OCRCR-SrSP, 
OCXCK.-SS^2 

OC.i.C'^ 
(occttxca^  . 
LUpr  <2RA0K) 

OCLCR-ri,  OCXjCR-SPSF. 
OCLC’X-SSSS 

PLUS 

CCcNYJ3,X> 

?mS-II,PJLU2-ISS, 
?tU2-2SI>7LtlS»S?SE, 
?Lt[2'S?SS  ,TLUjg-SSSP, 
?LXZ^-SSSS 

Table  •4,1 
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57 

5 
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1 
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4.2  SUMMARY  OF  THE  DATA  BASE 


Although  there  are  505  labels  listed  In  Chapter  3/  the 
piotual  number  of  elements  In  the  label  lists  for  each 
junction  will  be  larger  than  we  might  expect#  since  different 
permutations  of  labels  count  as  different  elements  In  some  of 
the  lists.  The  total  number  of  list  elements  needed  to 
represent  the  505  labelings  Is  717#  and  this  number  expands 
to  3256  when  the  region  Illumination  Information  Is  added  to 
the  labelings.  Table  4.2  shows  the  number  of  elements  In 
each  list  with  and  without  region  Illuminated  Information, 
This  table  differs  from  Table  2.2  In  that  It  Includes  only 
the  differences  In  the  list  lengths  which  are  caused  by 
adding  region  Illumination  Information. 


A  little  cleverness  Is  required  to  avoid  duplicate 
labelings  when  Including  the  different  permutations  of  X 
junctions.  This  Is  because  some  X  junctions  give  rise  to  two 
elements  In  the  X  labelings  list#  while  the  rest  add  only  one 
element.  Figure  4.3B  shows  an  X  junction  which  requires  two 
elements  to  be  added  to  the  list#  while  figure  4.3C  shows  two 
labelings  which  each  add  only  one  element  to  the  data  base. 
Most  shadow  X  junctions  give  rise  to  two  elements  In  the  data 
base#  and  most  junctions  without  shadows  give  rise  to  one. 
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It  is  now  possible  to  describe  how  the  program  handles 
each  junction  it  encounters: 

(1)  if  the  junction  is  an  I,  ARROW,  T,  K,  PEAK,  X,  KX, 
or  KXX,  it  uniquely  orders  the  junction's  line  segments  (by 
choosing  a  particular  line  segment  and  considering  the  rest 
as  ordered  in  a  clockwise  direction  from  this  line  segment). 


(2)  If  the  junction  is  a  FORK,  MULTI  or  XX,  it  chooses 
one  line  segment  arbitrarily. 


(3)  it  then  fetches  a  list  of  labels  which  contains 
every  possible  set  of  assignments  for  the  lines  (excluding 
the  possibilities  of  accidental  alignments  and  degeneracies, 
and  junctions  with  missing  lines)  and  associates  this  list 
with  the  junction. 


It  makes  absolutely  no  difference  whether  the  program 
obtains  this  list  from  a  table  (the  compiled  knowledge  case) 
or  whether  it  must  perform  extensive  computations  to  generate 
the  list  (the  generated  knowledge  case).  Similarly,  it  does 
not  matter  at  all  that  various  members  of  the  list  bear  a 
particular  relation  to  each  other,  e.g.  as  in  the  case  of  a 
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FORK  junction/  where  most  elements  of  the  list  have  two  other 
elements  which  are  permutations  of  the  element.  When  I 
return  to  the  Issues  of  degeneracies/  accidental  alignments 
and  missing  lineS/  all  I  need  to  show  Is  how  the  labelings 
corresponding  to  these  cases  can  be  added  to  the  appropriate 
junction  lists.  The  machinery  to  choose  a  particular  element 
operates  Independently  of  just  what  the  labelings  actually 
are. 


The  only  apparent  exceptions  are  those  labels  marked  to 
indicate  that  the  vertices  which  cause  them  are  either 
non-trihedral  or  concave  or  the  result  of  alignment  of 
surface  and  the  light  source.  This  information  can  be  used 
optionally  as  the  final  step  in  the  operation  of  the  program 
if  it  is  necessary  to  select  a  single  labeling  for  an 
ambiguous  junction,  in  such  a  case  these  marks  enable  the 
program  to  make  a  simple  judgement  about  which 
interpretations  are  most  likely.  Of  course  if  only  single 
interpretations  remain  before  the  final  step#  or  If  I  do  not 
care  that  some  junctions  are  not  uniquely  specified/  then  the 
program  does  not  need  to  use  these  heuristics  at  all.  (Such 
a  case  occurs  when  I  only  wish  to  find  edge  geometries  and  do 


not  care  about  region  illumination.  Often  ambiguous  labels 
differ  in  the  type  of  Illumination  for  various  regions  but 


.  , _ ilr"#"  - 
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provide  a  unique  labeling  of  edge  geometry.) 


2  123 
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5.0  SELECTION  RULES 

Now  that  I  have  shown  how  to  generate  a  large  number  of 
possible  labels  for  a  Junction/  I  will  show  how  to  go  about 
eliminating  all  but  one  of  them.  The  strategy  for  doing  this 
I nvolves: 

(1)  using  selection  rules  to  eliminate  as  many  labels 
as  possible  on  the  basis  of  relatively  local  information  such 
as  region  brightness  or  line  segment  directions/  and 

(2)  using  the  main  portion  of  the  program  to  remove 
labels  which  cannot  be  part  of  any  total  scene  labeling. 

5.1  REGION  BRIGHTNESS 

If  I  know  only  that  line  segment  L-A-B  Is  a  line  In  a 
scene/  then  it  can  theoretically  be  assigned  any  of  the  57 
possible  labels.  Once  I  know  that  L'A'B  has  an  ARROW  et  one 
of  its  ends  as  shown  in  figure  5. IB/  the  number  of 
possibilities  drops  to  19.  Suppose  that  I  knoW/  in  addition/ 
the  relative  brightness  of  R1  and  R2  In  the  neighborhood  of 
L-A-B  In  figure  5. 1C.  There  are  three  possibilities:  (1)  R1 
is  darker  than  R2/  (2)  R2  Is  darker  than  Rl/  or  (3)  the 
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brightness  of  R1  is  equal  to  the  brightness  of  R2. 

If  (1)  is  true,  i  know  for  certain  that  if  L-A-B  is  a 
shadow  edge,  then  R1  must  be  the  shadowed  side  and  R2  the 
illuminated  side.  Obviously  if  (2)  Is  true,  then  the 

opposite  holds,  I.e.  R2  must  be  the  shadowed  side  and  R1  must 

be  the  illuminated  side.  If  (3}  is  true,  then  It  is 

Impossible  for  L*A**B  to  be  a  shadow  edge  at  all.  (If  I  happen 

to  also  know  that  each  object  in  a  scene  has  all  its  '^aces 
painted  Identically  with  a  non-reflective  finish,  then  I  can 
also  eliminate  more  labels.  In  this  case.  If  (1)  is  true, 
then  L-A-3  cannot  be  labeled  as  a  convex  edge  with  region  R1 
illuminated  and  R2  shadowed  type  SS,  if  (2)  is  true,  then 
L'A'B  cannot  be  labeled  as  convex  with  R2  illuminated  and  R1 
shadowed  type  SS,  and  if  (3)  Is  true,  then  neither  of  these 
labels  is  possible.} 

5.2  SCENE/BACKGROUND  BOUNDARY  REVISITED 

It  is  easy  to  find  all  the  junctions  which  can  occur 
around  the  scene/background  boundary.  All  that  Is  necessary 
is  to  make  a  list  of  all  the  line  segments  which  can  occur 
along  the  boundary  and  then  look  for  segments  of  junctions 
which  are  bounded  by  two  members  of  this  set. 


f 
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E^ch  Junction  from  Chapter  3  which  has  a  star  In  one  of 
Its  segments  Is  listed  separately  from  junctions  which  have 
the  same  geometry  but  which  cannot  occur  on  the 
scene/background  boundary.  Thus  the  list  of  AROW  labels  Is 
divided  Into  ARROW'S^  a  list  made  up  of  those  labels  which 
can  occur  on  the  scene/background  boundary^  and  ARROW*!^  made 
up  of  those  which  must  occur  on  the  Interior  of  a  scene.  The 
total  list  of  junctions  which  can  also  appear  In  the  Interior 
of  a  scene  Is  found  by  appending  ARROW-B  to  ARROW” I,  since 
the  scene/background  labelings  can  appear  on  the  Interior  of 
the  scene  as  shown  in  figure  5.2.  Table  5.1  lists  the  number 
of  trihedral  junction  labels  which  can  occur  on  the  Interior 
and  on  the  scene/background  boundary  for  each  type  of 
junction.  Appendix  4  lists  all  of  the  junctions  which  can 
occur  on  the  scene/background  boundary  Including  region 
Illumination  information.  To  obtain  Appendix  4  I  have 
assumed  that  the  light  source  Is  positioned  In  one  of  the 
four  octants  of  space  above  the  support  surface.  This 
restriction  means  that  the  background  Is  guaranteed  to  always 
be  illuminated. 


l^GE 


Figure  5.Z 
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Obviously/  If  I  can  determine  which  lines  In  the  line 
drawing  are  part  of  the  scene/background  boundary/  this 
knowledge  can  be  used  to  great  advantage.  It  1 S/  In  fact/ 
not  difficult  to  determine  this  boundary;  any  of  several 
strategies  will  work.  Two  examples  are: 

(1)  Look  for  regions  which  touch  the  edge  of  the  field 
of  view  and  append  them  alt  together/  or 

(2)  Find  the  contour  which  has  the  property  that  every 
junction  lies  on  or  Inside  It  (see  Mahabala  1969). 

Doth  of  these  methods  require  that  the  scene  be 
completely  surrounded  by  the  background  region  or  regions. 

As  shown  In  figure  5.3,  method  (1)  works  even  If  the 
background  Is  made  up  of  more  than  one  region. 

Once  the  program  has  found  v/hich  region  Is  the 
background  region/  It  can  also  find  how  each  junction  Is 
oriented  on  the  scene/background  boundary.  Some  junctions 
always  appear  In  the  same  orientation;  for  example/  ARROW 
and  PEAK  junctions  can  only  be  oriented  so  that  the 
background  region  Is  the  region  whose  angle  Is  greater  than 
180  degrees/  and  K  junctions  can  only  have  the  region  whose 


TASK  131 


KI 


112 


BPGE  OF  \ 
FIEUP  OF  VIEV 


1?3 


FrcuKE  £5 

By  AWEirPIIOS  AJ-.L  THE  "REGIONS  "WHICH  TOUCH. 
THE  "EIGE  OF  THE  FIEEP  OF  Yra"W,  "WE  OBTAIH 
AEU  op  THE  BACKGRouttJJ  EXCETT  THE  SMAIU^ 
iRBGIOJiS  IW  ANO'RS".  By  FXNOIKG  AIUj  CON- 
TINUIKG  OOLiHHEAiE.  OE^CJUKED  EmB  S'BGPtENTS 
CQu-mANT'S  M.ATCSHEP  TS^  THBSE  KESIOHS  CAN 
SB  FOUNP  AMP  KP1>32I>  To  THE  BACRSR0U1«>  Al-SO. 


rtj-  "H'lrFffl'.*'': 


SECTION  5.2  132 

angle  Is  180  degrees  as  the  background  region  (see  Appendix 
4). 

Of  course  there  Is  no  way  to  easily  define  the 
orientations  of  FORK,  XX,  or  MULTI  junctions.  However,  as 
shown  In  figure  5.4,  the  L,  T,  X  and  KX  junctions  which 
appear  on  the  scene/background  boundary  can  be  sorted 
according  to  which  of  their  segments  Is  the  background 
region. 

Consider  figure  5.5.  Each  of  the  L,  T,  and  X  junctions 
Is  marked  to  Indicate  which  orientation  It  has.  Table  5.2 
shows  that  this  distinction  makes  a  significant  reduction  In 
the  size  of  the  starting  list  of  label  assignments  for  these 
junctions. 

5.3  EXTENDING  THE  SUPPORT  SURFACE 

Consider  a  problem  posed  by  the  scene  shown  In  figure 
5.6.  If  my  labeling  program  is  given  this  scene  with  the  set 
of  labels  defined  so  far,  the  program  will  not  find  a  unique 
labeling  for  L-C-D,  even  though  It  finds  L-A-B  to  be  a  shadow 
edge,  and  therefore  labels  R1  as  a  projected  shadow  region. 
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At  one  time  i  thought  that  I  would  need  to  write  a 
"demon"  program  which  would  check  for  shadow  edges  on 
the  table/  assert  that  such  a  shadow  region  is  copianar 
with  the  table/  and  then  eliminate  any  edges  other  than 
planar  ones  whenever  such  a  region  shares  an  edge  with 
the  illuminated  portion  of  the  table.  This  type  of 
approach  seemed  rather  ad  hoc  to  me/  and  started  me 
thinking  about  how  i  could  include  region  information  as 
part  of  each  junction  label.  There  could  be  many  added 
benefits  to  such  an  approach:  it  seemed  clear  that  just 
as  i  was  able  to  vastly  reduce  the  number  of  labels  from 
which  to  select  possible  ones  by  knowing  that  a  junction 
was  on  the  scene/background  boundary/  I  should  be  able 
to  reduce  the  number  of  labels  for  a  junction  which  was 
interior  to  tiie  scene  but  which  had  the  table  as  one  of 
its  region  segments. 


Therefore  I  defined  new  labels  as  shown  In  Table  5.3  to 
denote  any  edge  which  has  the  table  as  one  of  Its  adjoining 
regions.  Since  I  have  restricted  the  light  source  to  be  in 
the  quadrants  of  space  above  the  support  surface/  I  can  be 
certain  that  any  region  which  is  part  of  the  table  can  never 
be  self-shadowed/  type  SS.  i  have  used  this  fact  in 
constructing  Table  5.3.  Any  edge  which  touches  or  obscures 
the  trble  is  marked  by  appending  a  "T"  to  its  name  or 
printing  a  "T"  next  to  the  Une  segment.  The  old  labels 
without  "T"  are  understood  to  represent  edges  which  do  not 
have  the  table  as  either  of  their  adjoining  regions.  The 
addition  of  these  24  edge  types  brings  the  total  number  of 
line  labels  to  81. 
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The  tables  which  show  the  allowable  region  i 11 umi nat ion 
pairs  for  these  edges  (analogous  to  figure  2.6)  appear  in 
Table  5.4. 

To  update  the  lists  of  junction  labels/  I  must  add  to 
the  present  set: 

(1)  Ail  the  junctions  listed  in  Appendix  4/  but  with 
"T"  printed  next  to  both  line  segments  which  bound  the  region 
containing  the  star.  (These  regions  can  be  part  of  the 
background  of  the  scene/  i.e.  the  portion  of  the  table  which 
surrounds  the  scene  and  is  illuminated.)  Some  of  these 
junctions  can  also  have  other  projected  shadow  regions  which 
are  part  of  the  table/  so  that  '*T"  must  be  added  to  line 
segments  other  than  the  two  bounding  th*  starred  region 
These  junctions  are  listed  in  Appendix  5. 

(2)  All  the  junctions  wH{ch  can  bound  a  projected 
shadow  (type  SP)  region  which  is  also  part  of  the  table. 

Table  5.5  shows  the  situation  now  for  the  relative 
numbers  of  junctions  which  can  occur  on  the  scene/background 
boundary.  While  the  numbers  of  labelings  possible  If  the 
branches  were  labeled  independently  has  increased  sharply 
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with  the  ’ncrease  of  the  number  of  line  labels  from  57  to  81# 
the  actual  numbers  of  Junction  labelings  has  not  changed  for 
the  scene/background  boundary  and  has  Increased  only 
moderately  for  the  scene  Interior. 

The  value  of  these  additions  to  the  data  base  Is 
especially  pronounced  for  scenes  like  figure  1.2  where  the 
table  surface  accounts  for  seven  of  the  Interior  regions  as 
well  as  the  background  region,  in  addition  to  the 
improvement  for  scenes  of  this  sort#  there  are  other 
benefits.  Consider  figure  5.7.  How  many  objects  are  In  this 
scene? 

Now  look  at  figure  5.8.  Given  figure  5.7#  my  program 
will  return  both  interpretations:  the  one  we  would  usually 
expect  (region  R  as  the  table#  with  object  C  resting  on  the 
table)  or  the  Interpretation  shown  In  figure  5.8.  Thus  the 
new  labels  enable  the  program  to  make  finer  distinctions  than 
It  could  before.  Notice  that  we  could  also  use  the  table 
Information  to  make  another  heuristic  rule:  If  there  are  two 
Interpretations  of  an  Interior  region#  one  as  the  table  and 
one  as  an  extra  object#  choose  the  table  Interpretation. 

(This  corresponds  to  choosing  the  simplest  Interpretation# 
I.e.  the  one  with  the  fewest  objects.) 
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5.4  DISCUSSION 

This  section  Is  speculative;  nothing  In  It  Is  critical 
to  an  understanding  of  my  program. 

Underlying  the  previous  section  are  some  Important  kinds 
of  distinctions  between  levels  of  understanding  which  I 
believe  are  worth  pursuing  at  greater  length  at  this  point. 
There  are  several  levels  of  understanding  which  a  program  can 
have  about  a  particular  property  of  scene  features  (e.g. 

"this  region  Is  part  of  the  table"): 

(1)  the  first  level  of  understanding  Is  that  the 
program  be  able  to  express  the  fact  that  a  given  portion  of 
the  scene  does  or  does  not  have  the  property.  As  an  example^ 
until  the  program  had  the  labels  which  labeled  regions  that 
were  part  of  the  table#  It  could  not  express  the  difference 
between  the  two  possible  Interpretations  of  figure  5.7. 

(2)  The  next  series  of  levels  are  ones  where  the 
program  recognizes  more  and  more  Instances  of  features  which 
cannot  have  the  property  (and  consequently  recognizes  more 
precisely  where  the  property  can  apply).  My  program's  hard 
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knowledge  ends  at  this  level;  for  some  cases  Its 
understanding  Is  sufficient  to  uniquely  recognize  a  property/ 
In  other  cases  It  Is  unable  to  select  between  two  or  more 
possibl lltles. 


(3)  I  believe  that  the  next  levels  of  understanding  are 
characterized  by  the  iblllty  to  define  a  critical  test  (or 
series  of  critical  tests)  which  will  allow  a  pregram  to 
eliminate  remaining  possibilities  until  only  one  Is  left. 

Such  a  test  might  be  "If  I  remove  the  object  In  front/  I  v;lll 
be  able  to  see  whether  or  not  that  region  Is  connected  to  the 
table  surface"  or  "If  I  move  to  the  right/  and  If  that  region 
is  part  of  the  table/  then  I  should  be  able  to  see  an  edge  at 
point  (X/y)".  I  claim  that  this  must  be  the  next  level  of 
knowledge  since  many  line  drawings  simply  do  not  contain  cues 
which  allow  a  program  (or  a  person)  to  decide  between  various 
possibll ities. 


However/  let  me  make  a  distinction  between  knowledge  and 
expectation.  Even  if  I  am  not  allowed  to  make  further  testS/ 
I  still  expect  the  scene  to  have  a  particular  form. 

Moreover/  I  believe  that  this  expectation/  simulated  by 
heuristic  rules  in  my  program^  is  instrumental  In  deciding 
just  which  critical  tests  I  should  make.  For  example/  If  n 
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Interpretations  are  possible/  my  suspicion  Is  that  I  pick  the 
one  I  expect  to  be  true/  and  on  the  basis  of  this  expectation 
I  then  choose  a  test  (or  tests)  to  eliminate  all  the  (n-1) 
other  possibilities.  After  performing  this  test/  I  then  have 
knowledge  which  either  supoorts  my  expectation  or  forces  me 
to  form  or  choose  a  new  expectation. 

The  curious  fact  about  my  perception  is  that  I  only  see 
one  interpretation  at  a  time  even  when  I  know  that  a  scene  Is 
ambiguous.  (Take  for  example  the  reversing  Illusion  which 
alternates  between  a  vase  and  two  faces  in  profile/  depending 
upon  which  regions  are  viewed  as  figures  and  which  are  viewed 
as  background  (Koffka  1935)}. 

Even  when  I  have  Insufficient  solid  knowledge  on  which 
to  base  my  interpretation  of  a  scene/  my  expectation  seems  to 
carry  the  same  force  of  conviction  that  solid  knowledge 
would.  Nonetheless/  I  can  change  my  interpretations  of 
scenes  either  when  I  am  faced  with  new  evidence  (by  a  change 
in  my  relation  to  a  scene  or  change  in  the  scene)  or  if  I  am 
challenged  about  my  Interpretation  (Are  you  sure?). 
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Moreover/  I  am  aware  of  ambiguity  In  another  way;  even 
though  my  own  Interpretation  may  carry  a  sense  of  conviction 
with  It/  and  even  though  I  don't  usually  change  this 
interpretation  without  reason/  I  can  easily  understand  how 
another  person  could  Interpret  a  scene  In  one  way  while  at 
the  same  time  I  am  seeing  it  In  a  different  way/  where  I  am 
using  seeing  to  mean  interpreting  with  conviction  of  truth. 

i  do  not  believe  It  Is  worthwhile  to  delve  too  much 
deeper  Into  speculation  about  . Imilaritles  between  my  system 
and  human  perception.  For  example/  It  doesn't  seem  to  me  to 
make  much  sense  to  try  and  decide  whether  people  generate 
alternative  interpretations  when  they  are  needed  or  whether 
(as  In  my  program)  they  keep  all  the  active  alternative 
interpretations  but  are  only  aware  of  the  expected  one  at  any 
given  time. 

Nonetheless/  I  think  that  In  connection  with  ambiguity/ 
the  notion  of  knowledge  at  "other  levels"  as  the  ability  to 
eliminate  interpretations/  and  the  notion  of  expectation  as 
the  default  choice  of  an  Interpretation  when  I  run  out  of 
solid  knowledge/  are  ideas  of  central  Importance. 
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AN  EXAMPLE 

i  have  now  shown  how  to  use  selection  rules  to  narrow 
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6.0  THE  MA>N  LASELING  PROGRAM 

You  wMl  recall  that  5  described  at  some  length  In 
Section  2.4  a  "i liter  program"  which  systematically  removes 
junction  labels  whenever  there  are  no  possible  matches  for 
the  labels  Jt  adjacent  junctions.  Now  that  1  have  shown  a 
good  deal  mare  about  the  ^unction  labels  and  the  use  of  the 
selection  rules/  I  would  like  to  treat  this  program  again 
from  a  somewhat  different  perspective. 


6.’  A  SMALL  EXAMPLE 


B- 


I 


Suppose  that  the  program  Is  working  on  a  scene/  a 
portion  of  which  is  shown  In  figure  6.1.  Assume  that  the 
selection  rules  eliminate  all  labels  for  each  type  of 
junction  except  those  shown  at  the  bottom  of  the  figure. 
P.emember  that  the  selection  rules  operate  only  local  !>/  l.e. 
they  give  the  same  list  of  possibilities  no  matter  how  the 
labeling  has  proceeded  or  in  what  order  the  junctions  are 
taken.  All  the  step  numbers  refer  to  figure  6.2/  which 
summarizes  the  succeslve  lists  attached  to  each  junction; 
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Step  1:  Suppose  that  the  program  starts  with  si2,  and 
that  all  of  the  other  junctions  are  unlabeled.  Then  the 
program  assigns  list  L2  to  and  since  all  the  other 
junctions  are  uniabeled^  it  has  no  basis  on  which  to 
eliminate  any  of  the  labels  in  L2.  As  far  as  the  progt am 
knows/  all  of  these  lc.:/elings  are  still  possible. 

Step  2:  Now  suppose  that  It  next  labels  J1  by  attaching 
to  It  the  list  LI.  When  it  checks  the  junctions  adjacent  to 
J1  it  now  can  see  that  J2  has  already  been  labeled. 

Step  3:  Therefore  the  program  looks  at  J2  to  find  what 
restrictions/  If  any#  have  already  been  placed  on  line 
segment  L"Jl-d2.  In  this  case,  the  restrictions  are  that 
L-J1-J2  must  be  labeled  with  either  "B”  or  ”C"  or  "A*'  or  D" 
or  ”F"/  l,e.  with  any  letter  which  appears  third  in  an 
element  of  L2.  Each  element  of  LI  which  does  not  have  "B"/ 
"C"/  "A"/  "D"/  or  "F"  as  Its  first  letter  can  then  be 
eliminated.  Therefore  the  program  drops  "(G  H)"/  "(E  A)"  and 
"(E  B}”  as  possibilities  and  LI  becomes 

((A  B)  (A  C)  (A  0)  (3  B)  (B  E)  (C  F)  (F  A)). 
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Step  4:  Now  the  program  uses  this  same  reasoning  in  the 
opposite  direction.  In  what  ways/  If  any,  does  the  fact  that 
J1  must  be  labeled  from  the  list  restrict  the  labels  of 
adjacent  junctions?  Only  d2  of  the  adjacent  junctions  has 
been  labeled  so  far,  so  only  J2  can  be  affected.  The  only 
labels  which  are  possible  for  J2  are  those  elements  of  L2 
which  have  as  a  third  letter  "A”  or  "B"  or  "C"  or  "F". 
Therefore,  the  program  eliminates  "(F  A  D)"  as  a  possible 
label  and  L2  becomes 

((A  B  B)  (A  B  C)  (B  C  A)  (D  B  F)). 

Can  the  program  eliminate  any  other  labels  because  "(F  A 
D)"  has  been  eliminated?  No,  since  no  other  neighbors  of  J2 
except  J1  have  been  labeled,  and  the  reason  "(F  A  D)"  was 
eliminated  was  because  It  had  no  counterpart  at  Jl. 

Step  5;  The  program  now  can  move  on  to  J5  and  label  It 

with  L3. 

Step  6;  Each  label  for  J3  must  have  a  third  letter 
equal  to  one  of  the  first  letters  from  a  label  In  L2.  These 
letter  are  "A",  ''B"  and  "D".  Therefore  the  program 
eliminates  "(G  H  I )",  "(F  B  O”,  "(D  B  F)",  "(A  B  E)"  and  "(D 

C  G)"  from  L3  and  sets  L3  to  ((A  BA)  (B  C  A)). 


3^- V'-*r,«»j;f 


"XrXT^-TTT*^ 
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Step  7:  What  labels  now  are  possible  for  J2?  Since  the 
only  remaining  labels  for  J3  both  set  L'*J2-J3  to  "A",  the 
program  eliminates  "(B  C  A)”  and  **(D  B  F)"  from  L2  so  that  L2 
becomes  ((A  B  B)  (A  B  O). 

Step  8;  This  tlme^  a  neighbor  of  J2/  namely  Jl/  has 
been  labeled  al ready/  so  the  program  must  check  to  see 
whether  eliminating  the  element  of  L2  has  placed  further 
restrictions  on  LI.  Only  elements  of  LI  which  have  a  first 
letter  "B"  or  "C"  are  possible  labels  now/  so  the  program 
eliminates  "(A  B)"/  "(A  C)"/  “(A  0)"/  and  "(F  A)".  LI  thus 
becomes  ((B  B)  (B  E)  (C  F)). 

Since  no  other  neighbors  of  J1  are  labeled/  the  effects 
of  this  change  cannnot  propagate  any  further. 

6.2  DISCUSSION 

I  think  it  Is  easiest  to  view  the  process  of  the  program 


at  each  junction  as  having  three  actions: 
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(1)  attaching  labels/ 


(2)  removing  any  of  these  labels  which  are  Impossible 
given  the  current  context  of  this  junction/  and 

(3)  iteratively  removing  labelings  from  the  context  by 
allowing  the  new  restrictions  embodied  In  the  list  of  labels 
for  the  junction  to  propagate  outward  from  the  junction  until 
no  more  changes  in  the  context  can  be  made. 


There  are  two  points  of  Importance: 


(1)  The  solution  the  program  finds  is  the  same  no 
matter  where  It  begins  in  the  scene/  and 


(2)  the  program  is  guaranteed  to  be  finished  after  one 
pass  through  the  junctions/  where  it  performs  the  three 
actions  listed  above  at  each  junction. 


Given  a  line  drawing  with  N  junctions/  a  data  base  which 
has  no  more  than  M  possible  labelings  for  any  junction/  and  a 
situation  where  any  number  of  junctions  from  0  to  N  have 
already  been  labeled/  let  condition  C  be  one  where  for  each 
possible  line  label  which  can  be  assigned  to  a  line  segment 


el ther 


(1)  there  is  at  least  one  matching  line  label  assigned 
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to  the  junction  at  the  other  end  of  this  line  segments  or 
else 

(2)  the  junction  at  the  other  end  of  the  line  segment 
has  not  been  labeled. 

This  condition  C  must  be  satisfied  before  the  program 
moves  on  to  a  new  junction;  the  program  keeps  track  of  the 
line  segments  on  which  the  condition  may  not  be  satisfied. 

When  the  program  begins  labeling  a  junction  J/  assume 
that  C  holds  throughout  the  line  drawing.  When  the  junction/ 
previously  unlabeled/  has  labels  added/  the  only  line 
segments  along  which  C  can  be  violated  are  the  line  segments 
which  join  J  to  Its  neighbors/  and  it  is  possible  for  C  to  be 
unsatisfied  In  both  directions  oi>  these  segments  (l.e.  both  J 
and  J's  neighbors  may  have  unmatched  lint  labels). 

Therefore/  to  make  sure  that  the  program  needs  to  consider 
each  line  segment  a  minimum  number  of  timeS/  the  program 
first  uses  the  lists  of  possible  labels  specified  by  J's 
neighbors  to  eliminate  all  impossible  labels  from  J. 

To  see  why  this  is  the  correct  way  to  proceed/  suppose 
that  the  program  used  J's  initial  set  of  labels  to  eliminate 
some  labels  from  one  of  J's  neighbors/  Jl.  It  Is  then 
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possible  that  the  set  of  labels  for  J  can  be  reduced  further 
because  neighbor  J2  has  no  match  for  one  or  more  labels  still 
attached  to  J.  The  program  would  then  have  to  go  back  to 
line  L-J-Jl  again  to  see  whether  more  labels  could  be 
eliminated  from  Jl.  By  considering  the  effects  of  each  of 
J's  neighbors  on  J's  labels  first/  the  program  guarantees 
that  as  many  labels  as  possible  have  been  eliminated  from  J's 
label  list  before  using  this  list  to  recompute  the  lists  for 
J's  neighbors. 

Condition  C  can  now  only  be  untrue  along  line  segments 
joining  J  with  its  neighbors  and/  moreover/  can  only  be 
untrue  In  one  direction/  I.e.  J's  neighbors  may  have 
unmatched  labels/  but  not  vice-versa.  When  the  program 
eliminates  the  unmatched  labels  from  each  of  J's  neighbors/  C 
Is  now  satisfied  on  each  line  segment  joining  J  to  Its 
neighbors  and  C  can  only  be  unsatisfied  along  the  line 
segments  joining  J's  neighbors  with  the  neighbors  of  J's 
neighbors/  and  again  only  In  an  "outward"  direction/  I.e.  the 
junctions  two  line  segments  away  from  J  can  have  unmatched 
labels/  but  all  those  junctions  one  line  segment  away  (J's 
neighbors)  cannot  have  unmatched  labels. 
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The  line  segments  on  which  C  does  not  hold  continue  to 
spread  outward  to  the  neighbors  of  junctions  two  segments 
away  from  0/  then  junctions  three  segments  away  from  etc./ 
but  only  as  long  as  labels  are  being  removed  from  any 
junctions.  As  soon  as  the  program  reaches  a  step  where  no 
labels  are  removed  from  any  junction/  then  the  program  knows 
that  condition  C  must  be  satisfied  everywhere  In  the  scene/ 
and  it  can  move  on  to  the  next  unlabeled  junction. 

Figure  6.3  traces  a  situation  which  could  occur  on 
successive  steps  In  a  line  drawing  where  all  junctions  except 
J  have  been  labeled  already.  I  have  filled  in  the  line 
segments  along  which  condition  C  could  be  violated  at  each 
stage  of  the  program's  iterations.  The  mark  ">"  indicates 
which  junction  can  have  unmatched  labels;  it  is  used  like 
the  same  sign  meaning  "greater  than"/  so  that  you  can  read 

_ >  Sk 

as  "the  number  of  labels  at  Jl  is  greater  than  the  number  of 
labels  at  Jk"/  i.e.  Ji  may  have  labels  which  are  not  matched 
by  ones  at  Jk. 
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The  violations  of  C  can  spread  outward  to  eventually 
touch  any  line  segment  of  a  line  drawing/  but  only  If  the 
number  of  labels  can  be  reduced  at  each  junction  on  some  path 
between  the  junction  the  program  Is  currently  labeling  and 
the  line  segment.  If  any  of  the  junctions  In  Figure  6.5  were 
unlabeled  or  If  a  unique  label  had  already  been  found  for  the 
junction/  then  no  violations  of  C  could  propagate  through 
that  junction. 

Figure  6.4  represents  just  such  a  situation.  The  line 
drawing  Is  assumed  to  be  completely  labeled  except  for 
junction  J/  but  this  time  J1  already  has  been  uniquely 
labeled.  Thus  It  can  never  be  the  case  that  J1  has  unmatched 
labels.  Notice  that  Figure  6.4  also  represents  equally  well 
the  case  where  J1  has  not  yet  been  labeled. 

Cne  final  point;  the  process  Is  guaranteed  to  terminate/ 
since  If  there  are  N  junctions  and  no  more  than  M  labels 
possible  for  any  one  junction/  the  process  can  never  go  on 
for  more  than  M  x  N  steps  at  the  very  worst.  This  Is 
important  since  the  restrictions  can  propagate  back  to  the 
junction  which  Initiated  the  process.  To  see  that  the 
possibility  of  cycles  does  not  creaf any  difficulties/ 
consider  the  following  trick.  Supoose  that  as  soon  as  the 
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starting  Junction  has  been  checked  against  each  of  its 
neighbors/  that  all  the  remaining  labels  are  removed  from  it. 
The  restrictions  can  then  spread  outward  only  until  no  more 
changes  can  be  made;  now  look  at  the  process  as  though  the 
Junction  were  being  labeled  for  the  first  time  with  the  set 
of  junctions  just  removed  as  its  starting  junction  set.  This 
process  can  then  be  repeated  as  often  as  necessary/  but  the 
number  of  times  can  never  be  greater  than  the  initial  number 
of  labelings  assigned  to  the  junction/  since  the  process 
terminates  If  no  more  labels  can  be  removed  from  the  list  of 
possi bi 1 i t i es. 

6.3  CONTROL  STRUCTURE 

While  the  program  can  start  at  any  junction  and  still 
arrive  at  th j  same  solution/  the  amount  of  time  required  to 
understand  a  sc?:ne  does  depend  on  the  order  in  which  the 
junctions  are  labeled.  The  basic  heuristic  for  speeding  up 
the  program  is  to  eliminate  as  many  possibilities  as  early  as 
possible.  Two  techniques  which  help  accomplish  this  end  are 
to 

(1)  label  all  the  junctions  on  the  scene/background 
boundary  first/  since  these  have  many  fewer  Interpretations 
that  interior  junctions  dO/  and 
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(2)  next  label  all  junctions  which  bound  regions  that 
share  an  edge  or  junction  with  the  background. 

To  see  why  the  program  is  faster  when  It  eliminates  as 
many  possibilities  as  early  as  it  can,  I  must  first  give  some 
idea  about  the  amounts  of  computation  needed  for  various 
phases  of  the  program.  The  basic  operation  involves  removing 
unmatched  labels  from  junction  lists.  The  removal  is  done  in 
the  following  manner: 

Assume  that  the  junction  whose  list  cf  labels  must  be 
reduced  is  called  J2,  that  its  neighbor  is  Jl,  and  that  for 
any  label  in  the  lists  of  either  J1  or  J2,  the  first  line 
label  represents  the  line  joining  them.  Thus  if  (A  B  C)  is 
one  possible  junction  labeling  In  Jl's  list,  then  "A"  Is  the 
line  label  that  this  junction  labeling  would  assign  to  line 
L-J1-J2,  and  similarly,  if  sD  E  F)  is  a  labeling  from  J2's 
list,  the  "D”  Is  the  line  label  which  refers  to  L“J2-vll. 

Since  J2's  list  Is  the  one  to  be  reduced,  first  look  at 
Jl's  label  list  and  make  a  list  which  consists  of  the  labels 
which  J1  can  apply  to  L-J2-J1.  Notice  that  I  have  up  to  now 
glossed  over  the  fuct  that  for  most  lines,  the  label  appears 
different  depending  on  which  end  of  the  line  we  choose  as  a 
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reference  poInC,  Thus  If  line  L-J1-J2  Is  labeled 


S? 

then  from  Jl's  end  It  appears  to  be  labeled  as  "OCR-ISP"  and 
from  J2's  end  It  appears  to  be  labeled  as  ”OCL-SPI"  (for 
OClude  RIght-l 1 1 umlnated/Shadow-Projected  and  OCLude 
Lef t-Shadow-Projected/ i 1 1 uml nated  respectively).  Therefore 
what  we  really  want  Is  the  list  of  the  opposites  of  the  first 
elements  of  each  label  for  Jl.  Suppose  that  I  am  given  the 
scene  portion  shown  In  flgu;e  6.5.  If  Jl's  list  of  labelings 


((OCR*-!  I  PLUS- 1  I  OCL-I I  ) 
(OCR-ISP  PLUS-SPI  OCL-I I) 
(OCRM-II  PLUS- 1  I  OCLM-II) 
(5HCLR-ISP  OCR-SPI  OCLf^-ll)> 


Then  the  list  that  I  need  to  compare  J2's  labels  to  Is; 


LI  “  ((opposite  (OCR-1  I)) 
(opposite  (OCR-ISP)) 
(opposite  (OCRM-II) ) 
(opposite  (SHCLR-ISP))) 

«  (OCL-I I 
OCL-SP! 

OCLM-I  I 
SHCCR-SPI  ) 
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J2's  label  list  can  then  be  compared  to  LI;  the 
condition  which  must  be  satisfied  by  a  labeling  of  J2  in 
order  for  it  to  be  a  possible  labeling  is  that  the  line  label 
it  would  assign  to  L-Jl-21  be  a  member  of  the  list  LI. 
Continuing  with  this  example/  suppose  that  J2's  labeling  list 
i  s: 


L2  -  ((OCL-I I  OCH-I I  OCR-1  I) 

(OCL-ISP  OCR-SPI  OCR-i I) 
(OCL-ISS  OCR-SSI  OCR-f I) 
(OCL-SPI  OCR-1  I  OCR-iSP) 
(OCL-SPSP  OCR-SPI  OCR-ISP) 
(OCL-SPSS  OCR- 1  I  OCRM-M) 
(OCL-I  I  OCR-M  OCRM-li) 
<0CRM-ll  OCRCR-II  OCML-M)) 


Then  the  labeling  list  for  J2  after  comparing  L2  to  LI  is: 


L2'»  ((OCL-I I  OCR-1  I  OCR-1  I ) 

(OCL-SPI  OCR-1  I  OCR-ISP) 
(OCL-I  I  OCR-1  I  OCRM-ID) 


Now  I  return  to  the  original  claim/  that  It  is  desirable 
to  remove  as  many  labels  as  possible  as  early  as  possible. 
Suppose  a  junction  J  has  m*l  branches  and  n+q  labels/  and  in 
the  process  of  labeling/  q  of  these  labels  are  eliminated  by 
a  propagating  reduction  which  comes  in  on  one  of  J's 
branches:  this  requires  the  program  to  compare  n+q  labels  for 
members  in  a  list.  The  program  now  has  to  check  each  of  J's 
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branches  to  see  If  any  labels  for  adjacent  junctions  can  be 
removed.  Thus  it  must  compute  m  lists  analogous  to  LI  above 
and  each  of  these  lists  has  n  members.  Now  when  each  of 
these  lists  is  compared  to  the  label  lists  for  adjacent 
junctions/  the  program  must  make  an  average  of  n/2  tests  for 
equality  for  each  labeling  that  is  retained/  and  n  tests  for 
equality  for  each  labeling  that  Is  removed  (for  the  case 
where  it  looks  through  an  entire  list  and  finds  no  match). 
Therefore  for  each  portion  of  the  process  the  amount  of 
computation  involved  is  at  least  proportional  to  n. 

Because  the  amount  of  computation  is  at  least 
proportional  to  n/  it  is  undesirable  to  label  interior 
junctions  first/  since  most  of  these  have  much  larger  Initial 
values  for  n  than  do  scene/background  boundary  junctions. 

Not  only  does  it  take  more  computation  to  propagate  any 
reductions  through  these  junctions,  but  each  reduction  Is 
likely  to  be  smaller  as  well;  if  two  adjacent  junctions  can 
each  be  labeled  in  n  ways  out  of  a  total  of  N  theoretically 
possible  ways,  then  the  expected  number  of  labelings  they 
have  in  common  is  n  /N.  (This  number  Is  obtained  by  summing 
the  probaoility  of  a  match  for  each  of  the  n  labels  at  one 
junction;  thus 


.  -  —  _ 
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t=n 

/  (n/N)  «  n  X  (n/N)  -  n^/N.) 

Crl,Z,.... 

Typically  scene/background  boundary  Junctions  have  about 
1/10  the  number  of  possible  labels  an  interior  junction  can 
have/  so  that  the  expected  number  of  labelings  to 
scene/background  Junctions  will  have  In  common  Is  only  1%  of 
the  expected  number  for  two  interior  Junctions.  Similarly/ 
it  is  worthwhile  to  label  next  Interior  Junctions  which  are 
connected  to  Junctions  on  the  scene/background  boundary/ 
since  the  expected  number  of  labelings  In  common  for  these 
pairs  is  only  10%  of  the  number  for  interior  junctions. 
Finally/  as  I  mentioned  earlier/  it  Is  worthwhile  to  label 
all  the  Junctions  surrounding  regions  which  touch  the 
scene/background  boundary/  since  these  regions  contain  all 
the  "best"  kinds  of  Junctions/  and  because  a  chain  of 
Junctions  which  closes  on  itself  tends  to  be  far  more 
restricted  in  its  possibilities  tho*n  a  chain  of  the  same 
length  which  does  not.  (I  win  not  attempt  to  prove  that 
this  is  so;  I  think  it  is  fairly  obvious  that  the  effect  is 
true,  although  the  proof  of  the  effect  is  not.  It  is  much 
more  obvious  for  a  tree  search  procedure  than  for  this  one.) 
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I  Included  this  section  so  that  an  interested  readf 
could  get  a  better  feeling  for  the  operation  of  the  program 
and  also  to  suggest  some  ideas  for  extensions  of  this 
program.  For  example^  If  my  labeling  program  were  connected 
to  a  line-finding  program  such  as  Shlrai’s/  my  program  could 
be  adapted  to  provide  intelligent  guidance  for  deciding  where 
to  look  next  in  a  scene  on  the  basis  of  which  features  had 
already  been  found  (Shirai  1972). 

Another  Idea  which  might  be  interesting  to  follow  up  is 
a  possible  parallel  between  the  reasons  why  it  is  better  for 
my  program  to  start  on  the  scene/background  boundary  and  the 
observed  fact  that  people  presented  with  a  figure  on  a 
background  for  short  periods  of  time  see  detail  first  on  the 
figure/ground  boundary  and  require  longer  viewing  durations 
to  see  details  In  the  figure  interior  suggesting  that  our 
perception  proceeds  from  the  outside  inward  (Koffka  1935). 

I  mentioned  at  the  beginning  of  this  paper  that  the 
amount  of  time  (and  therefore  computation)  is  roughly 
proportional  to  the  number  of  line  segments  in  a  scene.  This 
may  not  seem  to  fit  with  the  obvious  fact  that  there  is 
really  nothing  to  prevent  the  effects  caused  by  labeling  a 
single  junction  to  propagate  to  evf-ry  portion  of  a  line 


/ 
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drawing. 

Ther«  are  good  physical  reasons  why  this  seldom  happens. 
The  basic  reason  is  that  some  Junctions  simply  do  not 
propagate  effects  to  ail  their  neighbors^  and  so  the  effects 
tend  to  die  out  before  getting  too  far.  The  prime  type  of 
junction  which  stifles  the  spreading  effects  is  the  T 
junction. 

In  most  T  junctions^  the  labelings  of  the  upright  and 
crossbar  portions  are  Independent.  Even  if  we  know  the  exact 
labeling  of  the  crossbar  portion  we  are  unlikely  to  be  able 
to  draw  any  conclusions  obout  the  labeling  of  the  upright  and 
vice-versa.  Since  objects  are  most  commonly  separated  by  T 
junctions^  the  effects  of  labeling  a  junction  are  for  the 
most  part  limited  to  the  object  of  which  the  junction  is  a 
part/  and  to  the  object's  shadow  edges,  if  any. 

Another  reason  why  effects  do  not  propagate  far  is  that 
when  junctions  are  unlabeled  or  when  they  are  uniquely 
labeled,  they  do  not  propagate  effects  at  all.  (This  reason 
was  illustrated  in  figure  6.4.)  Thus  when  few  junctions  are 
labeled  and  when  most  of  the  junctions  are  labeled  the 
effects  of  adding  restrictions  tends  to  be  localized. 
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6.4  PROGRAM  PERFORMANCE 

The  program  portions  i  have  now  described  are  adequate 
for  labeling  scenes  without  accidental  alignments^ 
non-tr Ihedral  vertices  or  missing  lines.  Within  this  range 
there  are  still  certain  types  of  features  which  confuse  the 
program^  but  before  showing  Its  limits^  I  will  show  some  of 
its  complete  successes.  In  all  the  scenes  that  follow^  I 
assume  that  the  program  knows  which  region  Is  the  background 
region^  and  that  It  also  knows  the  relative  brightness  of 
various  regions.  The  program  operates  nearly  as  well  without 
these  facts  but  not  as  rapidly.  Figure  6.6  shows  a  number  of 
scenes  for  which  the  program  produces  unique  labelings  or  Is 
only  confused  about  the  Illumination  type  of  one  or  two 
regions  (as  in  figure  6.60  and  6.61).  By  varying  someof  the 
region  brightness  values  or  omitting  them#  the  program  could 
also  be  similarly  confused  In  this  way  for  the  tops  of 
objects  in  figures  6.6A#  6.6B#  6.6E#  6.6G  and  6.6H.  In 
general#  the  program  is  not  particularly  good  at  finding  the 
illumination  types  for  regions  unless  the  regions  are  bounded 
by  concave  edges.  This  confusion  has  a  physical  basis  as 
well.  In  all  the  diagrams  I  have  drawn  these  top  surfaces  as 
though  they  were  parallel  to  the  table  so  that  the  should  all 
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be  labeled  as  type  i  ( M lumi nated}/  but  since  the  program  I 
have  described  so  far  uses  only  the  topology  of  a  line 
drawing/  It  has  no  way  of  distinguishing  the  scenes  I  have 
drawn  from  others  which  should  be  labeled  differently.  For 
example/  In  figure  6.7  I  have  redrawn  figures  6.6A  and  6.6B 
so  that  the  top  surfaces  are  type  SS  (Sel f*Shadowed)/  but  the 
figures  are  topologically  Identical. 

To  decide  whether  a  surface  Is  self -shadowed  or 
Illuminated/  one  must  be  able  to  associate  shadow  corners 
with  the  vertices  which  cause  them.  In  figure  6.7B/  If  C  Is 
caused  by  B/  then  the  top  of  the  block  Is  Illuminated/  and  If 
C  Is  caused  by  A  then  the  top  of  the  block  Is  sel f 'Shadowed. 
To  verify  that  A  causes  C,  place  a  straight  edge  on  the 
figure.  There  Is  an  Interesting  optical  Illusion  In  this 
figure;  It  appears  to  me  that  the  top  surface  of  the  block  In 
figure  6.8  should  be  type  SS/  but  In  fact  If  you  use  the 
straight-edge  test  I  described/  you  will  find  that  It  Is 
actually  Illuminated.  (I  did  not  put  In  any  shading/  to 
prevent  biasing  the  choice.) 

In  any  case/  I  think  that  the  Issue  here  Is  not  serlous/ 
since  the  program  still  finds  the  correct  edge  labels  for  all 
edges.  In  general  I  doubt  that  anyone  will  be  too  Interested 
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In  finding  the  Illumination  values  exactly;  in  the  program 
they  serve  primarily  as  labeling  aldS/  not  as  ends  In 
themselves.  However^  before  going  on  to  something  else,  I 
would  like  to  use  this  topic  to  illustrate  a  situation  i  have 
encountered  several  times  in  the  process  of  performing  this 
research.  I  noticed  early  in  my  study  of  scenes  that  If  all 
shadow  corners  and  their  causing  vertices  in  a  given  scene 
are  connected  by  straight  lines,  these  lines  have  roughly  the 
same  slope  throughout  the  scene,  provided  that  the  light 
source  Is  reasonably  far  away  from  the  scene  compared  to  the 
scene  size.  I  thought  that  this  fact  might  aid  me  a  great 
deal  in  finding  shadows.  What  I  did  not  see  was  that  until  I 
could  locate  shadows  and  their  causing  vertices,  I  couldn't 
connect  the  two  to  find  the  characteristic  slope;  but  if  I 
could  find  the  shadows  and  vertices,  then  I  knew  how  to  solve 
the  problem  already,  and  so  I  would  not  need  to  find  this 
slope  at  all!  There  is  at  least  one  type  of  case  where  this 
slope  is  important,  as  I  describe  In  the  next  section,  but 
for  the  most  part  the  topology  of  scenes  provides  adequate 
clues  for  finding  shadows. 
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6.5  PERFORI4ANCE  PROBLEMS 

Shadows  convey  a  considerable  amount  of  Information 
about  which  edges  of  an  object  touch  a  surface^  since  a 
shadow  edge  can  only  Intersect  the  edge  which  causes  It  If 
the  surface  the  shadow  Is  cast  on  touches  the  shadow-causing 
edge,  as  Illustrated  In  figure  6.SA.  As  long  as  shadows  are 
present,  a  program  can  find  relations  between  the  objects  In 
a  scene  and  the  background,  as  shown  In  figure  6.9B. 

However,  If  all  shadows  are  missing,  then  It  Is  Impossible  to 
decide  how  the  pieces  of  a  scene  are  related.  For  example  In. 
6.9C,  the  block  on  the  left  could  be  stuck  to  a  wall  or 
sitting  on  a  table  or  sitting  on  a  smaller  block vhich 
suspended  It  off  the  table;  there  Is  simply  no  way  to  decide 
which  of  these  cases  Is  true,  given  only  a  shadow-free  line 
drawing.  Moreover,  the  program  does  not  use  (at  this  point) 
knowledge  of  line  segment  directions  In  a  scene,  so  It  cannot 
even  distinguish  which  way  Is  up.  If  you  turn  figure  6.9C 
about  1/3  of  a  turn  clockwise,  there  is  a  reasonable 
interpretation  of  the  two  blocks  with  A  supported  by  B. 

Without  line  segment  direction  Information  the  program  finds 
all  these  interpretations  If  there  are  no  shadows. 
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In  figure  S.IOA#  each  of  the  segments  marked  with  a  star 
can  be  Interpreted  either  as  an  obscuring  edge  or  as  a 
concave  edge^  though  in  most  cases  choosing  one  or  the  other 
for  some  line  segment  forces  other  segments  to  be  Interpreted 
uniquely^  as  shown  In  figures  6.10B  and  6. IOC. 

As  in  the  previous  section^  there  are  scenes  which  are 
topologically  Identical  which  can  help  to  show  why  the 
program  finds  ail  these  labelings  as  reasonable 
interpretations.  Figure  6.11  shows  five  scenes  which  are 
topologically  identical  to  the  labeled  scenes  shown  in  Figure 
6. IOC;  In  each  of  these  scenes/  the  labeling  shown  seems  to 
me  to  be  the  most  reasonable  one  or  at  least  a  plausible  one. 

Figure  6.12  shows  the  next  problem  case.  Such  a  case 
occurs  when  we  can  see  only  enough  of  an  object  so  that  it  Is 
not  possible  to  tell  whether  the  region  is  a  shadow  or  an 
object.  If  i r  happens  that  the  ambiguous  region  is  brighter 
than  the  background  (or  what  would  be  the  illuminated  portion 
of  a  partly  shadowed  surface  of  the  feature  occurs  on  the 
interior  of  a  scene}/  then  the  program  can  eliminate  the 
possibility  that  the  region  Is  a  shadow.  Unfortunately/  if 
the  ambiguous  region  Is  darker  than  its  neighbor/  it  cannot 
tell  whether  the  region  is  a  shadow  region  or  a  dark  object. 
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In  figure  6.12/  do  you  think  that  both  A  and  B  should  really 
be  labeled  as  shadow  regions?  In  fact  neither  A  nor  B  can  be 
shadows!  You  can  prove  this  for  yourself  by  finding  the 
characteristic  light  source  slope  for  the  scene/  using  the 
front  object  and  Its  shadow.  Then  note  that  there  can  be  no 
hidden  objects  which  could  project  A  or  B.  Figure  6.13  shows 
this  construction.  It  Is  this  type  of  distinction  for  which 
the  light  source  slope  Information  could  be  useful. 

I  will  not  go  through  the  process  again  of  showing  how 
each  of  the  labelings  could  arise.  Clearly  the 
Interpretation  of  A  and  B  as  shadows  Is  reasonable  for  this 
scene/  since  I  can  easily  find  a  topologically  equivalent 
line  drav/Ing  where  some  obscured  objects  could  cause  the 
shadows.  The  program  needs  to  know  about  gravity/  support 
and  line  segment  directions  In  order  to  eliminate  some  of  the 
Interpretations  of  region  A.  Every  one  of  the 
Interpretations  Is  possible  for  3. 

A  closely  related  ambiguity  Is  Illustrated  in  figure 
6.14A.  Again  difficulties  arise  because  a  shadow-causing 
junction  Is  hidden.  The  fact  that  the  program  does  not  know 
at  this  point  about  gravity  can  be  visualized  as  meaning  that 
the  objects  which  form  both  sides  of  a  crack  can  appear 
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anywhere^  just  as  if  the  two  objects  were  glued  together. 
Figure  6.14B  shows  such  a  case. 

The  next  type  of  problem  involves  support  directly.  An 
example  of  this  type  of  difficulty  is  shown  in  figure  6.15. 

As  in  figure  6.10^  each  of  the  edges  which  is  ambiguous  is 
marked  with  a  star  i.'k  )  in  figure  6.15A/  and  the  possible 
label ingS/  both  "reasonable"  and  "unreasonable"  are  shown  in 
figures  6.15B  and  6.15C  respectively.  I  have  redrawn  figure 
6.15C  in  figure  6.16  to  show  scenes  with  the  same  topology 
which  have  what  were  previously  unreasonable  labelings  as 
their  reasonable  ones.  Actually  in  some  of  the  cases  I  have 
had  to  change  the  topology  slightly.  This  happened  because  i 
wanted  to  construct  an  example  which  contained  shadows  and 
which  exhibited  all  the  ambiguities  I  show  in  figure  6.15; 
while  1  was  not  able  to  easily  find  a  scene  which  satisfied 
these  criteria  and  also  did  not  require  changes  in  topology^ 
there  probably  are  such  scenes,  t  do  not  believe  that  any 
general  rules  can  be  derived  from  the  needed  modifications. 

One  final  type  of  ambiguity  is  interesting  and  also 
serves  to  emphasize  one  of  the  findings  of  the  work  reported 
in  this  chapter.  In  figure  6.17  I  show  the  two  types  of 
interpretations  my  program  returns  for  holes.  One  of  these 
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Interpretations  Is  the  one  I  expected;  I  was  surprised  that 
the  hole  was  ambiguous^  but  even  more  surprised  to  find  that 
I  had  missed  an  obvious  alternate  Interpretation  of  the  sane 
geometry.  The  alternate  Interpretation  shown  In  figure  6.17B 
does  not  even  need  to  be  dr;;;wn  with  different  line  segment 
directions  In  order  to  appear  reasonable. 

The  labelings  which  the  program  finds  must  be  made  up  of 
local  features/  each  one  of  which  Is  physically  possible/  but 
it  is  not  obvious  that  the  features  which  remain  should  each 
be  part  of  a  total  labeling  of  the  scene  which  Is  physically 
possible.  After  all/  the  only  conditions  I  Impose  are  that 
each  of  these  features  must  agree  with  at  least  one  other 
feature  at  each  neighboring  junction.  On  the  basis  of  the 
fact  that  the  main  labeling  program  does  not  leave  extraneous 
labels  on  junctions/  it  seems  clear  that  topology  provides  a 
major  portion  of  the  cues  necessary  to  understand  a  scene. 

in  the  next  chapter  I  show  some  heuristic  rules  which 
can  be  used  to  eliminate  some  of  the  labelings  which  people 
usually  consider  unlikely.  In  fact  the  true  case  is  that 
these  labelings  are  not  unlikely/  but  the  scenes  which  have 
these  labelings  as  reasonable  ones  <to  our  eyes)  do  not  often 
arise  In  our  experience.  Unfortuntely/  heuristics  sometimes 


I 


SECTION  6.5  200 


reject  real  Interpretations^  and  indeed  would  reject  e.nch  of 
the  Interpretations  shown  in  figures  6.11  and  6.16  In  favor 
of  the  ones  In  figures  G.iOB  and  6.15B.  Nonetheless/  in  the 
absence  of  solid  rules/  these  heuristics  can  be  useful.  In 


the  chapter  on  region  orientations  I  deal  with  the  types cf 
techniques  which  would  enable  a  program  to  find  the  labelings 
which  we  would  assign  to  these  line  drawings  without  resort 
to  heuristics. 
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7.0  NON-TRIHEDRAL  VERTICES  &  RELATED  PROBLEMS 

So  far  I  have  assumed  that  all  the  junctions  I  am  given 
are  normal  trihedral  junctions  and  essentially  that  the  line 
drawing  which  I  am  given  is  "perfect".  When  a  program  has  to 
be  able  to  accept  data  from  real  line  finders  and  from 
arbitrarily  arranged  scenes^  these  criteria  are  rather 
unreal i st ic. 

in  this  chapter^  I  show  how  to  correct  some  of  these 
problems  in  a  passive  manner*  By  passive  1  mean  that  the 
program  is  unable  to  ask  a  line  finding  program  to  look  more 
carefully  or  to  use  alternative  predicates  at  a  suspicious 
junction^  and  similarly  that  it  cannot  move  its  eye  or 
camera,  or  direct  a  hand  to  rearrange  part  of  a  scene  in 
order  to  resolve  ambiguities  (Gaschnig  1971). 

instead  l  handle  these  types  of  problems  by  including 
labels  for  a  number  of  the  most  common  of  these  junctions  in 
the  regular  data  base.  In  cases  where  the  program  confuses 
these  junction  labelings  with  the  regular  labelings  and  where 
I  want  a  single  parsing,  I  can  easily  remove  these  new  types 
of  junction  labels  first,  since  t  have  ir.  uded  special 
markers  for  each  labeling  of  this  type.  Moreover,  depending 
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on  the  reliability  of  the  program  which  generates  the  line 
drawing/  I  may  wish  to  remove  labels  in  different  orders. 

For  example/  if  a  line  finding  program  rarely  misses  edges, 
missing  edge  interpretations  can  be  removed  first;  if  a  line 
finding  program  tends  to  miss  short  line  segments/  then 
accidental  alignments  are  probably  being  generated  by  the 
program/  and  these  interpretations  can  be  retained  until 
last.  Therefore  the  labels  for  each  type  of  problem  are 
marked  with  different  indicators  In  the  data  base. 

7.1  NON-TRIHEDRAL  VERTICES 

Some  non^trihedral  vertices  must  be  included  in  the  data 
base;  indeed  some  are  much  more  common  than  many  of  the 
trihedral  vertices.  I  will  limit  the  number  by  including 
only  those  non-trihedral  vertices  which  can  be  formed  by 
convex  trihedral  objects. 

The  first  type  of  vertex  is  formed  by  the  alignment  of  a 
vertex  with  a  convex  edge  as  shown  In  figure  7.1  and  in 
figure  7.2.  In  figure  7.3  a  similar  set  of  junctions  is 
shown  for  objects  which  MARRY  (l.e.  have  coplanar  faces 
separated  by  a  crack  edge;  see  Winston  1970)  along  one  edge/ 
but  which  have  difference  face  angles. 


SECTION  7.1  207 


Figure  7.4  Illustrates  another  convnon  non-tr Ihedral 
vertex  which  results  again  from  objects  with  dissimilar  face 
angles.  This  time  I  need  a  new  type  of  edge,  a  separable 
convex  edge,  labeled  as  shown  In  figure  7.4. 

Figure  7.5  Illustrates  the  types  of  non-trlhedral 
vert  '^Ich  can  occur  when  one  block  leans  on  another.  In 

order  co  ..e-p  these  cases  from  being  confused  with  other 
trihedral  Junctions,  I  have  introduced  three  new  edge  types. 
These  types  only  can  occur  in  a  very  limited  number  cf 
contexts.  Figure  7.6  shows  some  of  the  ways  In  which  these 
edges  can  appear. 

In  the  data  base  each  of  the  labelings  shown  in  figures 
7.1,  7.2,  7.3,  7.4,  and  7.5,  and  any  other  junction  labels 
involving  the  leaning  edges  or  the  separable  convex  edges, 
are  marked  as  non^tri hedral .  Later,  if  I  wish  to  find  a 
single  parsing  for  a  scene  where  there  are  still  ambiguous 
labels,  removing  these  non-trlhedral  junctions,  if  possible, 
may  be  a  good  heuristic. 
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7.2  ACCIDENTAL  ALIGNMENTS;  FIRST  TYPE 

In  this  section  I  have  not  attempted  to  exhaustively 
list  every  possible  junction  labeling  which  results  from 
accidental  alignment.,  but  have  concentrated  on  Including  only 
the  most  common  cases.  There  is  some  justification  for  this. 
In  that  ambiguities  caused  by  accidental  alignments  can  be 
resolved  by  simply  moving  with  respect  to  the  scene. 

Figure  7.7  lists  all  the  junctions  which  can  take  part 
In  the  first  type  of  accidental  alignment  i  will  consider. 
This  type  of  alignment  occurs  v/hen  a  vertex  Is  closer  to  the 
eye  than  an  edge  which  appears  to  be  bu’*  Is  not  part  of  the 
vertex.  Thus  the  set  of  vertices  In  figure  7.7  are  exactly 
that  subset  of  the  scene/background  boundary  junctions 
(Appendix  4)  which  contain  only  obscuring  edges  on  the 
scene/background  boundary.  Figure  7.7  shows  only  those 
junctions  which  I  include  as  sufficiently  common.  The  rest 
are  excluded  because  they  Involve  unusual  concave  geometries 
like  those  found  in  SOMA  cube  pieces  (SOMA  cubes  are 
three-dimensional  puzzles  manufactured  by  Parker  Bros.  Inc./ 
Salem/  Mass.)  or  because  they  Involve  three-object  edges  or 
because  the  resulting  junction  would  have  enough  line 
segments  to  require  a  designation  of  "SPECIAL"  or  because  the 
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junction  would  require  the  alignment  of  the  eye  with  three 
points  in  space. 

There  is  no  regular  junction  which  could  be  confused 
with  any  of  tiJe  ARROW  or  K  junc\:ions  generated  by  the  the 
alignment  of  the  junctions  shown  In  figure  7.7  with  edges 
behind  them.  To  see  why  this  is  so#  con4^ider  figures  7.8  and 
7.9.  Figure  7.8  gives  names  for  the  distinguishable  region 
segments  for  each  type  of  junction,  7.9  shows  all  the 

K  and  ARROW  junctions  that  can  result  frotn  accidental 
alignment  with  each  each  of  the  junctions  shown  in  figure 
7.7*  Notice  that  the  background  region  can  only  appear  in 
segments  ARROWl#  ARR0W2#  Kl#  K2  and  K3  in  these  accidentally 
aligned  cases#  whereas  for  all  trihedral  ARROW  and  K 
junctions  which  can  appear  on  the  scene/background  boundary# 
only  segments  ARROWO  and  KO  (the  segments  of  these  junctions 
which  are  greater  than  180  and  equa'  to  180  degrees# 
respectively)  can  be  part  of  the  background.  Of  course  for 
the  junctions  where  no  segments  are  distinguishable  (e.g. 
forks)  or  where  the  junction  appears  on  the  interior  of  the 
scene#  these  accidental  alignment  cases  cannot  be  directly 
distinguished  from  the  regular  cases. 
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At  this  writing/  I  have  not  included  ail  these 
acc'dentai  alignment  types  in  the  program's  data  base/  but  i 
have  included  most  of  the  scene/background  boundary  cases  and 
a  number  of  the  interior  cases.  In  general/  I  have  assumed 
that  no  non-tri hedral  edges  or  three-object  edges  will  be 
among  those  obscured  since  both  the  alignment  itself  and  the 
edge  types  are  relatively  unlikely/  so  their  coincidence  at  a 
single  junction  is  extremely  unlikely. 

7.3  ACCIDENTAL  ALIGNMENT  WliHOUT  OBSCURING  EDGES 

Figure  7.10  shows  some  alignments  which  have  shown  up 
frequently  in  scenes  I  have  worked  with.  These  junctions 
have  occurred  because  (1)  our  line  finding  program  misses 
short  line  segments  (and  therefore  tends  to  include  more 
lines  than  it  should  in  a  single  junction)/  (2)  our  line 
finding  program  has  a  tolerance  angle  within  which  It  will 
call  edges  coll  inear/  so  some  edges  are  called  col  linear  even 
when  they  are  not/  and  (3)  edges  which  lie  in  a  plane 
parallel  to  the  surface  on  which  they  cast  shadows  are 
parallel  to  the  shadows  they  cast/  so  that  alignments  become 
particularly  likely  when  we  use  brlcks/  cubeS/  and  prisms. 
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Figure  7.11  shows  some  other  types  of  accidental  shadow 
edge  alignment  which  our  group's  line  finding  program 
frequently  yields;  these  junctions  are  relatively  common 
because  of  the  tendency  of  the  program  to  miss  short  line 
segments^  but  each  of  these  types  of  alignment  can  occur 
naturally  as  well.  (For  Information  on  our  line  finding 
program  see  Horn  1971  and  Shlral  1972.) 

7.4  ACCIDENTAL  ALIGNMENTS;  FINAL  TYPE 

The  worst  type  of  accidental  alignment/  In  terms  of  the 
number  of  new  junctions  It  can  Introduce/  occurs  v/hen  an  edge 
between  the  eye  and  a  vertex  appears  to  be  part  of  the 
vertex.  Fortunately/  all  of  the  types  of  junctions  which 
these  alignments  Introduce  are  either  Ks,  KAs  or  SPECIALS. 

To  see  why  this  is  so,  look  at  figure  7.12.  All  these 
labelings  can  be  quite  easily  generated  by  a  program  which 
operates  on  the  regular  data  base.  Notice  that  for  each 
obscured  vertex  labeling/  there  are  three  new  labelings 
generated/  since  the  near  region  can  have  any  of  the  three 
illumination  values. 
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Also  notice  that  any  of  these  junctions  which  appear  on 
the  scene/background  boundary  can  ocly  be  oriented  with  the 
backg'-ound  In  a  junction  segment  type  Kl/  K2^  K3/  KAl#  KA2,. 
KA3/  or  tCA4  (see  figure  7.8),  Therefore  It  Is  not  difficult 
to  recognize  the  cases  where  accidental  alignments  of  this 
type  occur  on  the  .scene/background  boundary  since  none  of  the 
regular  trihedral  junctions  can  ever  appear  on  the 
scene/background  boundary  In  any  of  these  orl>  ntat lens.  (The 
background  can  only  appear  normally  In  seg.neuts  of  type  KO  of 
KAO.) 


The  number  of  K  junctions  of  this  type  which  can  occ«jr 
Is  limited  by  the  fact  that  two  cf  tha  line  segments  tche 
col  linear  ones)  must  always  be  obscuring  edges  and  so  can  be 
labeled  in  a  total  of  108  different  ways  (Including  region 
Illuminations);  the  other  two  Une  segments  can  e^ich  be 
labeled  In  81  v;ays/  so  thera  can  be  no  mors  than  81x81x108  » 
708/588  possible  K  labelings.  In  fact/  as  usual/  there  are 
not  nearly  this  many  labelings.  To  find  the  limit  on  the 
number  of  these  junctions/  use  figure  7.12  and  Idble  5.3 
together,  as  shown  In  Table  7.1.  The  numbers  In  Table  7.1 
are  obtained  by  taking  the  total  number  of  Interior  labelings 
for  ,a  t  pe  of  junction  (remember  thot  this  number  Includes 
TABLE  labels  as  well)/  multlolylng  this  number  by  the  number 
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of  ways  In  which  It  can  form  a  K  junction/  and  multiplying 
this  number  by  three  (since  the  obscuring  region  can  have 
three  types  of  Illumination/  independent  of  what  the  other 
labels  are).  ThuS/  for  example/  there  are  109  ARROW 
label ingS/  and  each  can  be  used  two  ways  to  make  a  K  label  of 
this  type  (see  figure  7.12)/  so  the  total  number  of  K 
junctions  due  to  obscured  ARROWS  is  109x2x3  >  654.  Each 
ARROW  labeling  can  be  used  In  only  one  way  to  form  a  KA 
junction/  so  the  total  number  of  these  Is  109x1x3  *  327. 

While  I  could  include  these  labelings  directly  In  the 
data  base/  their  number  is  clearly  unwieldy,  in  any  event/  I 
managed  to  find  a  way  to  include  the  labelings  exactly  but  In 
a  manner  somewhat  different  than  those  I  have  been  dealing 
with  so  far.  In  order  to  show  this  method/  I  first  have  to 
fill  In  some  gaps  I  left  earlier. 

7.5  MORE  CONTROL  STRUCTURE 

In  this  section  I  return  again  to  the  main  labeling 
program  and  describe  what  happens  when  the  program  Is  unable 
to  label  a  scene  consistently/  using  the  set  of  labels  with 
which  it  has  been  equipped. 
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The  program  is  written  In  MICRO-PLANNER,  a  programming 
language  with  automatic  back-up  facilities  (Sussman  et  a1 
1971).  Before  the  program  begins  labeling  a  junction  J,  it 
saves  the  context  of  the  junction  (l.e.  the  labeling  which 
existed  before  the  program  assigned  any  labels  to  J).  As  the 
program  iteratively  eliminates  the  labels  which  can  now  be 
removed  because  of  the  new  constraints  which  J  adds,  it 
checks  at  each  step  to  make  sure  that  at  least  one  label 
remains  possible  for  each  line  segment.  If  this  number  ever 
goes  to  zero  for  any  line  segment,  the  program  assumes  that  J 
is  the  source  of  the  problem,  l.e.  that  J  needed  a  label  that 
was  not  in  the  list  assigned  to  it  by  the  selection  rules. 
When  this  happens,  the  program  restores  the  context  to  the 
state  that  existed  before  it  began  labeling  J,  and  It  marks  J 
to  indicate  that  J  cannot  be  labeled  from  the  normal  label 
lists.  Once  J  has  been  marked  in  this  manner.  It  appears  to 
neighboring  junctions  to  be  just  1 1 ke  a  junction  which  has 
not  been  labeled  yet,  and  therefore  J  Imposes  no  conditions 
at  all  on  the  possible  line  labels  for  Its  neighbors.  The 
program  can  then  continue  and  as  long  as  two  adjacent 
junctions  are  not  left  unlabeled  at  the  end  of  the  program's 
operation,  every  line  segment  can  be  assigned  a  value  or  set 
of  values,  just  as  if  every  junction  had  been  labeled. 
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The  problem  with  this  arrangement  Is  this:  suppose  that 
the  program  Is  given  a  line  drawing  which  has  one  junction 
that  cannot  be  labeled  from  the  regular  set  of  junction 
labelings.  Clearly  If  the  program  labels  this  junction  last/ 
It  will  be  unable  to  label  the  junction  and  will  give  the 
correct  result.  However/  If  this  junction  Is  labeled  before 
any  of  Its  neighbors/  then  It  Is#  of  course/  automatically 
assigned  labels  from  the  normal  set/  for  none  of  the 
surrounding  junctions  impose  any  constraints  on  It.  In  this 
case/  one  or  more  perfect’. y  normal  junctions  In  the  scene 
will  eventually  be  marked  as  unlabelable/  and  the  resulting 
total  labeling  for  the  scene  will  be  Invalid.  In  general/  if 
the  bad  junction  Is  labeled  toward  the  end  of  the  program's 
operation/  then  the  total  scene  labeling  Isorrect/  and  If 
the  junction  Is  labeled  early  In  the  program's  operation/  the 
total  scene  labeling  Is  Incorrect. 

My  first  attempt  at  solving  this  problem  was  to  label 
all  Ks  and  KAs  last.  In  many  cases  the  Ks  and  KAs  were  then 
Indeed  correctly  Identified  as  unlabelable  from  the  normal 
set.  Hovtever/  I  managed  to  come  up  with  a  much  neater 
solution  which  enables  the  program  to  generate  labels  for 
these  otherwise  unlabelable  junctions. 
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As  before/  I  have  the  program  label  all  Ks  and  KAs  last/ 
but  this  time  i  modified  the  labeling  procedure,  if  a 
junction  cannot  be  labeled  from  the  normal  set/  Instead  of 
marking  it  unlabelable  t  generate  possible  labelings  by 
modifying  the  line  drawing  so  that  it  contains  equivalent 
junctions  which  are  not  accidentally  aligned/  and  then  t 
label  these  junctions  in  the  normal  manner.  Thus/  as  shown 
in  figure  7.13/  if  the  normal  set  of  junctions  is  inadequate 
to  label  a  K/  the  most  reasonable  alternative  is  that  the 
junction  is  actually  an  obscured  L  vertex.  Therefore  ! 
change  the  line  drawing  (saving  the  original  of  course)  and 
try  to  label  the  new  line  drawing.  This  change  is  equivalent 
to  moving  the  eye  slightly  to  see  what  type  of  junction  is 
obscured/  except  that  since  the  program  is  unable  to  move  its 
eye  and  therefore  does  not  know  what  the  real  vertex  type  iS/ 
it  keeps  trying  various  alternatives  until  one  works/  or 
until  it  hits  a  default  case.  In  the  example  shown/  the 
program  finds  a  reasonable  ir.terpretat ion  on  the  first  try. 


If  it  had  not/  then  the  program  would  next  have  tried  to 
label  the  junction  as  an  obscured  ARROW/  since  ARROWS  are  the 
next  most  common  type  of  junction  after  Ls. 
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Notice  that  the  condition  for  a  modification  to  be 
reasonable  is  not  as  simple  as  the  old  condition  for  a  single 
junction/  as  illustrated  in  figure  7.14.  The  condition  for 
figure  7.14A  is  that  J/  Jl/  and  J4  must  all  be  labeiabie. 
Before  there  was  no  condition  joining  J1  and  J4;  if  they  do 
not  match  now/  it  does  not  matter  whether  JO  can  be  labeled 
or  not  because  a  total  labeling  would  be  impossible.  This 
means  that  the  program  has  to  be  able  to  save  the  context 
until  it  has  finished  checking  the  labeling  of  several 
junctions/  and  that  it  should  only  finalize  the  modifications 
when  it  has  proved  that  every  portion  of  the  new  line  drawing 
is  reasonable.  To  illustrate  further/  in  figure  7.14B  I  show 
the  modifications  necessary  to  interpret  JO  as  an  obscured 
ARROW  junction.  These  modifications  create  a  new  junction/ 
and  the  two  junctions/  JO  and  JJO  must  both  be  checked; 
unless  both  can  be  labeled  consistently  this  interpretation 
is  impossiblec 

In  fact/  !  can  carry  this  Idea  even  further.  Suppose 
that  a  K  junction/  JO/  is  actually  an  accidental  alignment/ 
but  that  since  other  K  and  KA  junctions  in  the  line  chawing 
have  not  yet  been  labeled  JO  can  be  labeled  from  the  normal 
sot  of  labelings,  later  another  K/  which  should  be  labeiabie 
from  the  normal  set  cannot  be  labeled/  since  the  wrong  choice 
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was  made  for  JO.  To  eliminate  this  type  of  difficulty/  I 
require  all  K  and  KA  junctions  to  agree/  and  If  they  do  not 
agree/  the  program  can  back  up  to  any  of  the  K  and  KA 
junctions  until  It  has  actually  tried  every  combination  of 
interpretations  for  the  junctions.  Thus  the  program  should 
not  finalize  any  of  the  labels  for  K  or  KA  junctions  until 
all  of  them  agree. 

This  solution  is  still  not  guaranteed  to  contain  the 
correct  one;  the  program  will  be  satisfied  with  the  first 
set  of  modifications  for  the  K  and  KA  junctions  which  gives  a 
complete  labeling.  To  be  certain  of  including  the  correct 
solution/  the  program  would  have  to  try  every  combination  of 
interpretations  for  every  K  and  KA  and  save  all  the  ones 
which  give  complete  labelings.  Eventually  I  hope  to  Include 
this  ability  when  I  modify  the  program  to  run  in  t  a  CONNIVER 
language  (McDermott  &  Sussman  1972);  this  language  has 
better  facilities  for  developing  and  saving  parallel 
contexts/  whereas  MICRO-PLANNER  does  not.  MICRO-PLANNER  is 
oriented  toward  a  tree  search  model  of  problem  solving  where 
the  branches  of  a  solution  tree  are  explored  until  a  correct 
solution  is  found,  in  my  case/  the  problem  is  that  therenay 
be  more  than  one  correct  solution. 
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In  any  case^  when  I  programmed  this  ability/  I  lumped  a 
number  of  Junction  types  together  Into  a  default  case  for  two 
reasons:  this  lessened  the  possibility  of  stopping  before 
getting  the  desired  ("correct”)  solution/  and  Itenabled  the 
program  to  run  much  faster  and  required  a  much  smaller 
program  than  would  have  been  needed  If  I  had  Included 
separate  machinery  for  each  type  of  junction.  The  program 
tries  the  possibilities  for  a  K  In  the  following  order: 

(1)  try  to  label  the  K  from  the  normal  label  lists. 

(2)  try  to  label  the  K  as  an  obscured  L  vertex. 

(3)  try  to  label  the  K  as  an  obscured  ARROW  vertex. 

(4)  If  all  these  fall/  label  the  K  as  two  T  junctions 

(see  figure  7.15). 

The  default  condition  represents  the  exact  opposite  of 
the  previous  conditions.  The  two  Ts  result  If  Instead  of 
moving  the  eye  (by  Imagination)  to  see  what  vertex  Is  behind 
the  obscuring  edge/  the  program  moves  Its  eye  (by 
Imagination)  to  completely  cover  the  vertex  and  eliminate  the 
accidental  alignment.  Notice  that  the  default  condition 
gives  much  weaker  constraints  than  could  be  obtained  by 
trying  all  the  rest  of  the  junction  types  explicitly.  The 
only  relation  that  must  hold  for  the  two  T  uprights  Is  that 
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the  region  between  them  (marked  R  in  figure  7.15)  have  an 
illumination  value  which  matches  both  uprights.  Nonetheless 
this  is  a  much  stronger  condition  than  is  imposed  by  leaving 
the  Junction  totally  unlabeled  and/  in  addition^  the 
coll  inear  segments  (L-A-B/  L-B-C/  L-C-D  in  figure  7.15)  can 
all  be  labeled  unambiguously  as  occluding  edges.  The 
information  I  throw  away  requires  that  the  two  uprights  be 
adjacent  segments  of  the  same  vertex#  where  this  vertex  can 
presumably  be  labeled  from  the  normal  label  lists. 


7.6  MISSING  EDGES 


I 


Missing  edges  usually  occur  when  the  brightness  of 
adjacent  regions  is  nearly  the  same#  since  most  line  finding 
programs  depend  heavily  on  steps  in  brightness  to  define 
edges.  I  have  made  no  attempt  to  treat  missing  edges 
systematically#  but  have  only  Included  a  few  of  the  most 
common  cases  in  the  data  base.  Clearly  missing  edge  junction 
labels  could  be  systematically  generated  by  a  program  merely 
by  listing  all  possibilities  for  eliminating  one  edge  from 
each  junction  label.  This  procedure  would  generate 
(n-l)x(old  number  of  regular  labels)  for  each  junction  type 
(where  n  Is  the  number  of  line  segments  v/hich  make  up  the 
junction)#  and  clearly  this  would  be  a  rather  unmanageable 
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number  of  new  labels.  The  number  of  new  labels  could  be 
lessened  somewhat  by  noting  that  certain  types  of  edges  such 
as  cracks  are  likely  to  be  missed  whereas  certain  other  edges 
such  as  shadows  are  relatively  unlikely  to  be  missed. 

Even  if  a  program  such  as  mine  can  recognize  that  a 
junction  must  be  labeled  as  having  a  missing  edge/  problems 
still  remain  about  exactly  how  the  line  drawing  should  be 
completed.  This  difficulty  is  illustrated  in  figure  7.16. 
Depending  on  the  line  segment  directions  and  lengths/  the 
missing  edge  junction  0  can  be  connected  to  vertex  A/  vertex 
B  or  vertex  C/  even  though  the  topology  of  all  the  line 
drawings  is  identical. 

The  missing  edge  junctions  which  are  included  in  the 
program's  data  base  are  all  L  junctions  which  result  from 
deleting  one  of  the  branches  of  a  FORK  junction  with  three 
convex  edges,  incidentally/  in  the  examples  shown  in  figure 
7.16/  my  program  finds  each  of  the  given  interpretations/  but 
finds  no  other  interpretations/  I.e.  it  finds  no 
interpretations  which  do  not  involve  missing  edges. 
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A  rule  which  can  be  helpful  In  removing  Impossible 
missing  edge  interpretations  is  that  if  a  region  Is  bounded 
by  only  one  junction  which  can  be  interpreted  as  having  a 
missing  edge  In  that  region^  then  that  missing  edge 
interpretation  is  Impossible.  (There  must  be  another 
junction  to  connect  with  the  missing  edge.)  A  similar  rule 
depends  on  including  the  label  that  the  missing  edge  would 
have  had  in  each  missing  edge  labeling.  In  this  case^  the 
rule  is  that  not  only  must  there  be  a  pair  of  missing  edge 
junctions  around  a  region  In  order  for  either  of  them  to  be 
possible/  but  this  pair  must  also  match  in  the  label  that 
each  gives  to  the  missing  edge.  One  final  rule  is  that  the 
previous  rules  only  hold  If  the  pair  of  missing  edge 
junctions  are  not  adjacent  to  one  another  (I.e.  each  pair  of 
junctions  can  be  connected  by  only  one  straight  line). 

If  more  than  one  edge  is  missing/  then  a  program 
requires  greater  constructive  understanding  than  my  program 
haS/  although  I  believe  that  there  are  reasonably  simple 
rules  which  allow  a  program  to  solve  scenes  even  If  they  are 
as  bad  as  the  one  shown  in  figure  7.17.  For  example/  Shirai 
has  demonstrated  that  the  silhouette  of  a  scene  contains  a 
great  deal  of  information  about  where  interior  lines  and 
junctions  can  appear  (Shirai  1972).  Although  he  does  not 
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consider  scenes  with  shadows^  I  believe  that  the  same 
principles  which  he  uses  are  applicable  for  shadowed  scenes. 
Freuder  has  also  written  a  sophisticated  heuri st Ic  progran 
which  fairly  reliably  fills  in  edges  missed  by  our  group's 
line  finding  programs  (Freuder  1971a/  1971b). 

7.7  HEURISTICS 

As  I  have  mentioned  earlier  in  several  places/  the 
program  is  able  to  remove  junction  labels  selectively 
according  to  a  crude  probability  measure  of  the  relative 
likelihood  of  various  individual  feature  interpretations. 
These  heuristics  are  a  poor  substitute  for  foolproof  rules; 
in  essence  I  view  the  heuristics  as  an  expedient  method  for 
handling  problems  I  have  not  yet  been  able  to  solve  properly. 
As  I  explained  in  Section  5.4/  these  heuristics  may 
nonetheless  be  of  considerable  value  in  guiding  programs 
which  find  sound  solutions. 

There  is  not  much  to  say  about  the  heuristics 
themselves.  The  ones  I  am  using  currently  lump  all  the 
"unlikely"  junction  labels  into  one  clasS/  the  "likely"  ones 
into  another/  and  simply  eliminate  all  the  "unlikely"  labels 
as  long  as  there  are  "likely"  alternatives. 
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However  there  are  some  Interesting  cases  where  I  have 
found  that  I  can  usually  eliminate  the  unwanted  the  problem 
scenes  In  Section  6.5.  Obviously/  to  solve  these  cases 
exactly  would  require  a  great  deal  more  programming  effort. 

Heuristic  1:  Try  to  minimize  the  number  of  objects  In  a 
scene  Interpretation. 

implementations : 

(1)  Make  shadow  L  Junction  labet:,  (see  figure  7.18A) 
more  likely  than  any  other  type  of  L  junction. 

(2)  Make  labels  representing  Interior  TABLE  regions  nwre 
likely  than  the  equivalent  labels  that  do  not  involve  TABLE 
regions. 


(3)  If  regions  can  be  interpreted  either  as  shadows  or 
as  objects/  make  shadow  interpretations  more  likely. 


Heuristic  2:  Eliminate  Interpretations  that  have  points 
of  contact  between  objects  or  between  objects  and  the  TABLE 
unless  there  Is  solid  evidence  of  contact. 


Figure  718 
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Implementation:  Make  ARROW  junction  labels  which  have 
two  concave  edges  and  one  convex  edge  (see  figure  7.16R)  less 
likely  than  ARROW  labels  of  other  types. 

These  heuristics  select  Interpretations  (1>/  (2),  and 
(7)  from  figure  6.10/  Interpretations  A(l)  and  B(2)  from 
figure  6.12/  Interpretation  (1)  from  figure  6.14/  and 
interpretations  (1)/  (2)/  (3)/  (4)/  (5)/  and  (9)  from  figure 
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8.0  REGiCN  ORIENTAViONS 

What  has  obviously  been  ntssing  fron)  all  that  I  have 
shown  sc  far  Is  a  connection  between  line  segment  directions 
on  the  retina  and  possible  labelings  for  these  lines.  Such  a 
connection  Is  extremely  useful  If  the  program  Is  to 
understand  gravity  and  support.  In  this  chapter  I  describe 
approaches  to  this  problem  which  i  have  not  yet  Included  In 
my  program.  There  Is  probably  as  much  work  required  to 
properly  add  the  ability  to  handle  direction  Information  as  I 
have  already  Invested  In  my  program.  Nonetheless^  I  believe 
that  this  chapter  provides  a  good  Idea  of  the  work  that  needs 
to  be  done  as  well  as  the  physical  knowledge  that  these 
additions  will  allow  one  to  Include  In  the  program. 

8.1  LINE  LABEL  ADDITIONS 


I 


I 


To  begin  with,  I  Investigate  the  partitioning  of  each 
edge  type  into  three  subtypes,  a  technique  analogous  to  the 
ones  I  used  earlier  to  divide  concave  edges  Into  four  classes 
and  all  edges  Into  types  according  to  their  region 
illumination  values.  As  In  the  case  of  occludi ng  edges,  the 
line  values  are  only  defined  with  respect  to  a  reference 
point  and  direction,  where  the  usual  reference  points  are 
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junctions.  The  three  values  are: 

(1)  U  (Up)  -  an  edge  directed  up#  away  from  the  TABLE. 
The  reference  end  is  closer  to  the  TABLE  than  any  other 
points  along  the  edge  In  the  reference  direction. 

(2)  0  (Down)  “  directed  downward  toward  the  TABLE.  This 
is  the  opposite  of  U#  the  same  edge  but  with  the  other  end  of 
the  line  and  the  opposite  direction  on  tbo  line  as 
references. 

(3)  P  (Parallel)  -  parallel  to  the  TABLE  or  in  the  plane 
of  the  TABLE. 

Notice  that  there  are  some  immediate  limitations  that 
can  now  be  set  on  the  set  of  junction  labelings: 

(1)  Any  shadow  edge  or  concave  edge  marked  with  a  "T"# 
l.e.  which  is  in  the  plane  of  the  table#  automatically  can 
have  only  one  direction#  P#  in  this  partitioning. 

(2)  Any  junction  which  has  one  or  more  shadow  and 
concave  edges  labeled  "T”  must  have  its  edges  of  other  types 
in  the  U  direction#  since  the  edges  at  such  junctions  must 
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either  be  In  the  plane  of  the  TABLE  or  above  this  plane. 

(3)  Two  edges  which  bound  the  same  region  and  which  are 
parallel  or  coll  Inear  must  both  have  the  same  direction 
value^  U,  or  P.  This  fact  can  be  chained  through  several 
regions. 

Figure  8.1  illustrates  these  facts;  U  Is  Indicated  by 
placing  an  arrow  along  the  side  of  a  line  segment  pointing  In 
the  Up  direction. 

Notice  that  these  rules  also  allow  a  program  to  find 
horizontal  surfaces,  an  Important  part  of  the  notion  of 
support,  A  horizontal  surface  can  be  defined  In  this  system 
of  notation  as  any  region  bounded  by  two  or  more  edges  which 
are  both  marked  P  ( j|  )  and  which  are  not  parallel  to  each 
other  or  col  II  near.  Moreover,  any  edges  which  are  In  the 
plane  of  a  horizontal  surface  can  then  also  be  marked  as 
parallel  to  the  TABLE,  regardless  of  the  directions  of  these 
lines  on  the  retina.  Finally,  any  junctions  which  bear  a 
relationship  to  a  horizontal  surface,  analogous  to  the  one 
that  I  mentioned  earlier  for  junctions  which  had  segments  In 
the  plane  of  the  TABLE,  can  similarly  have  their  other 
segments  labeled  U.  Figure  8.2  Illustrates  these  points. 
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These  rules  are  not  particularly  helpful  when  there  are 
no  parallel  edges;  !t  Is  possible  to  chain  some  values  In 
the  absence  of  parallel  edges  and  horizontal  surfaces/  but 
generally  such  chaining  cannot  be  carried  very  far. 

Depending  on  the  way  that  edges  deviate  from  parallel/  it  is 
sometimes  possible  to  assign  an  Up  direction.  (In  some  of 
the  figures  which  follow  I  have  not  marked  the  lines  with 
their  normal  labels/  but  have  only  included  the  dl'-ectlon 
labels  for  clarity.)  See  figure  8.3/  and  note  that  since 
edge  L-A-B  Is  not  parallel  to  L-C-D/  I  can  mark  L-C-D  with  an 
Up  direction  as  shown.  This  means  that  since  L-E-F  Is 
parallel  to  L-C-D/  It  can  also  be  marked  with  an  Up 
dl rection. 

8.2  AN  EXAMPLE 

Using  the  methods  I  have  already  discussed  plus  one 
other  piece  of  new  information/  I  can  show  how  to  eliminate 
some  labelings  for  a  line  drawing  if  I  know  the  line  segment 
directions.  To  see  how  these  can  helP/  consider  again  the 
example  I  showed  In  figures  5.10  and  5.11/  as  Illustrated  now 
in  figure  8.4A.  Because  L-A-B  Is  parallel  to  L-C-D  and  L-B-E 
is  parallel  to  L-D-F/  R1  must  be  horizontal/  assuming  that 
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I  the  labeling  shown  Is  true.  By  the  same  kind  of  reasoning  R? 

»A 

I  must  be  horizontal.  Now  the  additional  rule  Is:  two 

I 

I  horizontal  regions  can  only  be  separated  by  crack/  shadow  or 

obscuring  edges.  Therefore/  the  labeling  shown  is 
impossible/  since  L-C-G  Is  a  concave  edge,  and  consequently 

>  cannot  separate  two  horizontal  regions.  Similarly  I  can 

>  eliminate  the  labeling  shown  In  figure  8.43. 

8.3  GENERAL  REG  I  ON .ORI ENTAT! ONS 


In  this  section  I  define  a  quantization  scheme  which 
assigns  to  each  visible  region  one  of  sixteen  values.  The 
regions  are  named  In  as  sensible  and  simple  a  manner  as  1 
could  devise/  and  are  defined  with  respect  to  a  coordinate 
system  which  Is  Itself  defined  by  the  TABLE  surface  and  the 
position  of  the  eye  viewing  the  scene.  The  region 
orientation  values  are  each  shown  In  figure  8.5;  I  assume 
that  this  figure  will  serve  as  an  adequate  sped  f  I  cation  for 
the  meaning  of  the  different  orientation  values.  If  the 
scene  Is  moved  with  respect  to  the  eye  or  vice-versa,  then 
the  region  values  (except  Table  and  Horizontal)  may  cnange, 
and  regions  previously  invisible  may  become  visible.  Thus 
the  region  orientation  values  are  not  Inherent  properties  of 
.the  surfaces,  but  are  only  defined  with  respect  to  a 
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particular  eye-table  arrangement. 

If  a  region  K1  is  type  FRV  (Front  Right  Vertical)  and  an 
edge  separating  tnis  region  from  region  R2  Isa  shadow  edge# 
then  region  R2  must  also  be  type  FRV  (see  figure  8.6A).  The 
problem  Is  not  quite  so  simple  when  the  other  edge  types  are 
Involved.  To  give  the  flavor  of  what  I  would  like  to  be  able 
to  do  In  general^  note  that  If  an  edge  separating  R1  and  R2 

Is  vertical  on  the  retina,  and  R1  appears  to  the  right  of  R2 

'0 

on  the  retina,  then  R2  can  only  be  type  FLV  or  type  FV  or 
type  FRV  (see  figure  8. SB), 

8.4  GENERAL  LINE  DIRECTiONS 

Before  I  can  carry  out  this  type  of  association  In 
general,  I  must 

(1)  define  line  directions  on  the  retina  and 

(2)  define  line  directions  In  the  scene  domain  with 
greater  precision,  and 

(3)  show  how  to  fr;d  the  scene  direction  values,  given 
a  labeled  line  drawing  and  the  retinal  line  directions. 
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Throughout  this  chapter  I  assume  that  the  eye  Is  far 
enough  away  from  the  scene  so  that  vertical  edges  In  the 
scene  project  Into  North/South  lines  on  the  retina.  Since 
the  definition  of  North/South  edges  Includes  a  tolerance 
angle  ,  the  eye  does  not  need  to  be  at  Infinity  for  this 
condition  to  hold.  By  the  same  reasoning  I  assume  that 
parallel  edges  can  be  recognized  without  resort  to 
perspective  or  vanishing  point  considerations. 

First  I  define  the  retinal  line  directions  In  terms  of 
compass  points  as  shown  In  figure  8.7. 

Next  /  In  figure  8.8/  I  define  the  names  for  the 
directions  of  lines  In  the  scene  by  showing  examples  for  each 
.•  type  possible  direction.  These  names  resemble  the  names  for 
region  orlentat Ions,  but  1  will  always  use  lower  case  letters 
In  referring  to  the  line  names  and  will  use  upper  case 
letters  when  I  refer  to  the  region  names. 

Now  to  make  the  connections  between  the  retinal  and 
scene  line  directions/  note  that  I  can  catalog  all  the 
possible  edge  directions  In  the  scene  domain  which  can  map 
into  each  of  the  fllrectlon  values  on  the  retina.  As  an 
example  of  how  to  do  thIS/  In  figure  8.9  I  show  all  the  edge 
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directions  possible  for  an  edge  which  bounds  a  tvpe  FRV 
region.  The  diagrams  In  this  figure  show  that  an  NE 
(Northeast)  line  on  the  retina  which  bounds  a  type  FSV  region 
can  be  an  edge  of  types  bru^  brp^  or  brd,  that  an  E  (east) 
line  on  the  retina  which  bounds  a  type  FRV  region  can  only  be 
caused  by  a  type  brd  edge/  etc.  Table  8.1  Is  a  summary  of 
the  types  of  scene  edges  which  can  cause  lines  of  each  type 
on  the  retina#  arranged  according  *5  the  types  of  regions 
that  each  edge  can  bound. 

Now  to  tie  everything  together#  notice  that  an  edge  can 
only  separate  two  regions  If  the  edge  could  have  the  same 
direction  In  both  regions  bounding  the  edge.  Therefore#  to 
find  all  the  region  pairs  that  an  N  (North)  edge  (as  seen  on 
the  retina)  could  separate#  look  down  the  N  column  In  Table 
8.1  and  find  all  the  pairs  of  regions  which  can  share  an  edge 
which  points  In  a  particular  direction.  A  north  pointing 
edge  can  thus  separate  any  of  the  following  pairs  of  region 
types  (this  Is  not  a  complete  list); 

((TA  TA)  (H  H)  (TA  LU)  (H  LU)  (H  RU) 

(RU  TA)  (RU  H) 

(FRV  FRV)  (FRV  FLV)  (FRV  FV) 


(FLV  FRV)  (FLV  FV)  (FLV  FLV) 


TABUS  8.1h>iREcrroN  ori-iNi:  onk^it.ina. 


PASE  2£>S 


N  NE  E  SE  S  SW  W  NW 


H  orTA  ht-F  *'P  f*"?  fp  ■flp  Ip  kip 


FU 


FY 


FRU 


FRY 


FRD 


EU 


BRU 


FLU 


FLY 


FfD 


bra  rp  ■krd  "fd  "fid  Ip  blu 


vu  vru  rp  vrd  vd  vtd 


■fa  fru  rp  brd  bd 


■u 

'P  br<l 

-a 


bru 

VU  brp  b'^'d  Va 


bmsi  brd  bd 


P  VtsI 

brd  bi-^  '^'^d  -fp  'Piu.  -flu 
fvci  ^ 


brd,vr<i  -  «  > 

bd  bt-d  brd  -fu  -flu.  ’flu. 

fi-u. 


brd  rp  -fru  -fbc  -flu.  i  Ip 


•frtv  fru,  "fu 


bp  I  vru  -fru  “fru.  -fp 
•fru 


bru  . 

bu  Y'-u  tru  frp  -i-d 

1  -fru  frd 


„  "Fru 

fru  ■f'^p  vd 
-frd 


r  r  P  fru, frp 

tu.  Tr'‘<  Tr'U  frd.vni  bd 

bird 


•flu 

vlu 

blu 


flu,  V  lu 
blu,blp 
bia 


fid 

vld  bid 
bid 


■fid 

Ylci,  bid 


bid  bid 


SECTION  R.4  2S9 


(FV  FV)  (FV  FLV)  (FV  FRV) 

(LU  H)  (LU  RU)  (LU  LU ) 

(RU  H)  (RU  LU)  (RU  RU) 

(BLU  BRU)  (BRU  BLU)) 

Not  all  these  pairs  can  be  separated  bythe  same  types 
of  edges;  shadows  and  cracks  can  only  separate  regions  with 
the  same  orientation  values^  and  convex  edge  pairs  become 
concave  edge  pairs  If  the  order  of  the  pairs  Is  reversed. 

For  example#  a  North  line  separating  regions  with  orientation 
values  (FLV  FRV)  represents  a  convex  edge  (where  the  ordering 
of  the  regions  Is  In  a  clockwise  direction)#  but  If  the 
orientation  values  are  (FRV  FLV)  for  a  North  line#  this  must 
represent  a  concave  edge.  This  fact  Is  Illustrated  In  figure 
8.10. 

If  the  Up/Down/ Paral 1  el  designations  are  also  included 
In  the  regular  labeling  program,  then  It  Is  possible  to  make 
even  finer  distinctions.  Table  8.2  shows  some  of  the  lists 
of  region  orientation  pairs  which  can  be  assigned  to  lines 
having  the  Indicated  labels  and  directions. 
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A  program  can  use  this  Information  In  the  following 

ways: 

(1)  If  there  are  ambiguities  remaining  after  the 
regular  labeling  program  has  finished^  pick  a  single 
labeling/  assign  region  values  using  the  lists  shov/n  In  part 
In  Table  8.2/  and  see  whether  this  labeling  can  represent  a 
possible  Interpretation;  If  the  Interpretation  Is  not 
possible/  then  the  program  will  be  unable  to  assign 
orientation  values  to  every  region/  very  much  like  the  case 
earlier  In  this  chapter  where  a  concave  edge  could  not 
separate  two  horizontal  surfaces. 

(2)  Region  Illumination  values  can  be  tied  In  with  the 
region  orientation  values.  For  example/  If  a  scene  Is  lit 
from  the  left/  and  the  light-eye  angle  Is  less  than  90 
degrees  (see  figure  3.11;  the  llght-eye  angle  I s the  angle 
between  the  projections  of  the  eye  and  the  light  onto  the 
plane  of  the  TABLE/  as  measured  from  the  center  of  the 
scene)/  then  a  region  cannot  be  labeled  simultaneously  as 
orientation  type  FLV  and  Illumination  type  SS  (Self- 
Shadowed). 
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(3)  AH  these  facts  provide  a  neat  way  to  Intej^ratp 
stereo  information  Into  a  scene  description.  For  example,  as 
shown  In  figure  8.12,  If  an  edge  Is  truly  vertical  (type  vu) 
then  It  must  appear  as  N  <No;th)  In  any  retinal  projection  of 
a  stereo  system.  However  an  edge  which  Is  of  type  hp  (back 
parallel)  can  appear  to  be  N  on  the  retina  because  of  the 
particular  placement  of  the  eye  with  respect  to  the  scene. 

If  the  eye  Is  shifted  slightly  to  the  right,  this  edge  will 
now  appear  to  point  NE  (Northeast)  and  If  the  eye  Is  shifted 
to  the  left/  the  edge  will  appear  to  point  NV<  (Northwest). 
Clearly  this  knowledge  would  enable  a  program  to  much  more 
severely  restrict  the  region  orientation  pairs,  and 
consequently  the  labelings,  that  can  be  assigned  to  a  line 
drawing  of  a  scene,  without  the  region  and  edge  orientation 
formalisms  (or  other  sMilar  formalisms)  It  Is  not  possible 
for  a  program  to  understand  this  stereo  Information,  although 
ore  could  undoubtedly  find  ad  hoc  ways  of  using  the 
I nformat Ion. 

(4)  All  the  possibilities  for  region  orientations  can 
be  generated  by  the  function  I  called  ILLUMINE  In  Section 
4.1,  For  each  labeling  which  vhe  program  finds,  ILLUMINE  can 
select  region  pairs  according  to  the  line  directions  and  line 
'  hels,  and  build  up  a  set  of  region  orien  atlon  values  In 
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exactly  the  same  manner  that  ILLUMINE  builds  up  sets  of 
region  Illumination  values.  The  difference  Is  that  therp  are 
far  too  many  region  orientation  values  In  general  to  possibly 
include  them  In  precompiled  form;  the  values  must  be 
generated  from  the  greatly  reduced  set  of  possibilities  that 
remain  after  the  regular  labeling  program  has  completed  Its 
work.  The  reason  why  there  are  so  many  possibilities  Is  that 
there  are  so  many  possible  region  orientations.  Each  edge 
can  potentially  have  16x16  =  256  region  orientation  pairs  as 
opposed  to  the  nine  possible  region  Illumination  pairs. 

8.5  SUPPOKT 

Using  the  region  orientation  values,  I  can  nov/  define 
the  set  of  edges  along  which  support  must  hold,  the  set  of 
edges  along  which  support  can  hold,  and  the  set  of  edges 
along  which  support  cannot  hold.  By  support  1  mean  what  Is 
commonly  termed  either  resting  on  or  leaning  on. 

To  start  with,  1  can  eliminate  from  consideration  any 
edges  which  are  shadows,  convex  edges,  obscuring  edges,  or 
concave  edges  made  up  of  one  object  or  of  three  objects,  and 
i  can  say  for  certain  that  support  is  exhibited  along  any 
concave  edge  which  has  the  TABLE  as  a  bounding  region.  In 
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addition  edges  labeled  as  "leaning"  (see  Section  7.1)  point 
to  places  where  support  relations  must  hold,  although  support 
does  not  hold  along  the  leaning  edges  themselves,  since  these 
are  either  obscuring  or  convex  edges. 


I 
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The  Impc.-rant  fact  Is  that  these  edges  exhibit  support 
regardless  of  their  directions  on  the  retina,  so  that  there 
Is  no  problem  with  edges  such  as  L-A-B  In  figure  8.13.  The 
best  previous  rules  to  find  where  support  holds  In  a  scene 
(see  Winston  1970)  are  not  able  to  handle  cases  like  this; 
Winston's  rules  were  biased  toward  finding  AKROWs,  Ks  and  Xs 
which  have  vertical  (or  at  least  upward  r^ofntlng)  lines  as  do 
all  of  the  cases  In  figure  8.14  (this  figure  Is  a  copy  of 
figure  2-41  from  Winston  1970).  In  addition,  Winston's  rules 
failed  to  find  one  support  relation  for  the  leaning  block; 
his  rules  assumed  that  objects  would  be  supported  by  face 
contact  only. 

Although  my  program  can  find  support  In  cases  like 
figure  8.13,  It  Is  Important  to  note  that.  In  general.  It  Is 
not  possible  to  use  my  regular  labelings  and  I'ne  directions 
alone  to  find  which  edges  exhibit  support  and  which  do  not. 
Suppose  that  on  the  basis  of  the  frequency  of  crack  edges 
like  the  ones  shown  In  figure  8.15A  I  decided  to  label  as 
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SupportIng/crack  edges  ones  In  which  the  arrow  of  the  crack 
label  points  SW,  V'l^  or  Nw^  and  to  class  all  the  others 
together  as  being  crack  edges  without  support  relations. 

Then  in  figure  8.1'iB  edges  L-B«C  and  L-C-D  would  be  correctly 
marked  but  L-A-B  would  not.  I  could  patch  up  the  rule  by 
saying  that  if  support  holds  for  one  non-col  1 1  near  line  in  an 
X  Junction  it  must  hold  for  the  other  non-co 1 1 i near  line  of 
the  X  as  well.  Unfortunately  this  rule  causes  the  program  to 
assert  that  support  holds  between  the  two  objects  in  flg(ire 
8.15C#  since  support  would  be  transferred  by  the  rule  from 
L-B-C  to  L-A-B. 

Similarly,  for  concave  edges  I  cannot  use  line 
directions  and  the  direction  of  the  arrow  on  the  label  to 
define  support.  As  an  example,  observe  that  while  L-A-B  In 
figure  8.15D  does  not  exhibit  support,  L-C-D  in  figure  8.15E 
does. 

Region  orientation  values  can  help  to  avoid  these 
problems,  at  least  for  some  cases.  (There  are  some,  cases 
such  as  the  one  in  figure  8.15F  ,  where  I  do  not  kn'"**  whether 
to  say  that  support  holds  along  L-A-B  and  L-B-C  or  not.) 
Interestingly  enough,  with  region  orientations  specified,  I 
do  not  necessarily  need  line  directions,  aithouf’li  I  certainly 
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need  line  directions  to  find  the  region  orientation  values  to 
begi n  with. 

An  example  of  an  edge  where  support  must  hold  is  any 
concave  edge  which  has  a  horizontal  surface  on  its  left  when 
one  looks  along  the  edge  in  the  direction  of  its  "arrow",  as 
does  L-C-l)  in  figure  8.15E. 

Some  examples  of  edges  where  support  cannot  hold  are 
concave  edges  which  have  vertical  surfaces  (FRV,  FV,  or  FLV) 
or  downward  pointing  surfaces  (FRD,  FD,  or  FLD)  on  the  left 
of  the  edges  when  looking  along  the  direction  of  the  "arrow"; 
line  L-A-B  in  figure  8.15D  is  an  edge  of  this  type. 

While  I  do  not  show  how  to  do  so  here,  I  believe  that 
the  best  way  to  add  the  understanding  of  support  to  the 
framework  of  my  program  is  to; 

(1)  add  support  labels  to  linos  in  junction  labelings 
where  support  can  hold,  and  ad(i  these  labelings  to  the 
regular  set  of  labels. 
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(2)  as  usual/  do  nochins  whan  a  line  raprasents 
unambiguously  a  support  edge  or  an  edge  without  support/  and 

(3)  when  there  Is  ambiguity/  use  the  region  orientation 
values  to  help  decide  the  issue*  To  do  thiS/  note  that  since 
there  is  a  connection  between  the  edges  which  can  h^va 
support/  line  directions/  and  region  orientations/  I  can  us^ 
the  function  ILLUMINE  again  to  eliminate  Impossible 
combinations  and  hopeful y  decide  where  support  can  and  cannot 
hold. 


1  have  no  great  confidence  that  such  a  system  will  show 
where  support  must  hold  for  c»irtain/  but  the  knowledge  about 
where  support  can  hold  combined  with  the  knowledge  that  every 
object  must  be  supported  somehow/  should  allow  the  program  to 
do  quite  well.  I  suspect  that  the  program  will  be  quite  good 
at  finding  places  where  support  cannot  possibly  hold.  To 
solve  these  problems  fully  a  program  needs  considerable 
knowledge  about  stability/  gravity/  and  friction.  These 
problems  are  outside  the  scope  of  this  paper;  for  a 
discussion  of  some  of  the  Issues  involved  see  Blum  et  a1 
1970. 
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To  give  a  feeling  for  the  number  of  new  Junctions  which 
would  be  required  In  the  data  base/  1  have  shown  some 
.••inctlons  In  figure  8.16  which  can  Involve  support.  Figure 
8.16A  illustrates  the  fact  that  any  concave  edge  which 
touches  the  TABLE  must  be  ^  support  edge.  In  figure  8.168/ 

If  the  crossbar  of  the  T  (the  col  linear  lines)  exhibits 
support  on  one  of  Its  halves/  then  It  must  exhibit  support  on 
the  other  half  as  well  and  the  support  direction  must  be  the 
same  for  both  of  these  edges.  Similarly/  In  figure  8.16C 
both  non-co 1 1 Inear  edges  must  have  the  same  support  or  lack 
of  support  values.  If  each  of  the  branches  which  can 
potentially  exhibit  support  relations  were  labeled 
Independent! 7/  then  the  cases  In  figures  8,16A  and  8. IBB 
would  each  have  27  possible  support  assignments  Instead  of 
three  and  nine  respectively/  and  the  case  In  figure  8.16C 
would  have  9  assignments  Instead  of  the  actual  three.  Thus 
the  same  kinds  of  techniques  which  I  have  shown  earlier  for 
other  descriptions  would  almost  certainly  work  well  for 
support  cases  too.  Finally/  obscuring  edges/  which  have  up 
to  now  accounted  for  the  biggest  Increases  In  the  numbers  of 
new  labels  when  the  old  labels  were  split  Into  subtypes  do 
not  even  taKe  part  In  this  partitioning/  so  that  the  Increase 
In  the  total  number  of  labelings  should  be  well  vithin 
bounds. 
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Figure  8.16 
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Of  course  my  present  program  can  already  list  lines 
where  support  may  hold  (I.e.  all  crack  and  two-object  concave 
edges)/  and  as  before/  simple  heuristics  would  allow  the 
program  to  say  with  some  confidence  where  support  could  or 
could  not  hold.  Clearly/  it  would  also  be  quite  natural  to 
call  some  of  the  support  assignments  In  figure  8.16  "likely" 
and  certain  others  "extremely  unlikely". 


9.0  HISTORICAL  PERSPECTIVE 


It  Is  Instructive  to  reexamirie  earlier  vision  work  which 
dealt  with  similar  problems  In  the  light  of  the  formalisms  I 
have  presented  In  this  paper.  In  this  chapter  I  review  the 
work  of  Guzman  (Guzman  1968)#  Rattner  (Rattner  1970),  Orban 
(Orban  1970),  Freuder  (Freuder  1971a,  1971b),  Dowson  (Oowson 
and  Waltz  1971,  Dowson  1971a,  Dowson  1971b),  Huffman  (Huffman 
1971),  and  Clowes  (Clowes  1971,  Clowes  et  al  1971). 

In  what  follows  I  hope  to  give  you  some  appreciation  for 
the  real  advances  In  thinking  about  vision  which  were  brought 
about  by  these  authors.  Ten  years  ago  the  whole  area  of 
computer  vision  was  uncharted  territory,  and  It  was  certainly 
far  from  obvious  where  one  should  begin.  Today,  while  there 
are  innumerable  questions  still  unanswered,  we  have  some 
definite  ideas  about  how  vision  systems  could  be  organized 
and  about  the  reasons  why  many  appealing  systems  such  as 
perceptrons  and  template  matching  schemes  are  Inadequate 
models  for  vision  systems  (Minsky  &  Papert  1970). 
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9.1  GUZMAN’S  SEE  PROGRAM 

Guzman's  work  Is  probably  the  most  famous  of  the  earlier 
vision  work/  and  indeed  his  approach  was  a  dramatic  departure 
from  what  had  been  done  before  him.  His  formal  i<yns  were 
designed  to  group  regions  together  into  bodies.  Basically 
his  program  did  this  by  identifying  each  line  in  a  line 
drawing  as  linking  or  not  linking/  where  linking  means  that 
the  regions  on  both  sides  of  the  line  belong  to  the  same  body 
and  not  linking  means  that  there  Is  no  evidence  about  the 
line;  It  may  be  either  linking  or  the  regions  on  either  side 
of  the  line  may  belong  to  different  bodies.  Guzman  used  a 
set  of  junction  types  exactly  as  my  program  does  (L/  ARROW/ 

T/  etc.)  but  he  Included  only  one  labeling  for  each  type  of 
junction.  Guzman's  junction  set  Is  shown  in  figure  9,1. 

There  can  be  two  conditions  for  jny  line  In  a  line 
drawing  after  the  labeling.;  have  been  assigned  to  each 
j  unct i on: 

(1)  the  labels  at  either  end  of  a  line  agree/  In  which 
case  the  labels  are  assumed  to  be  correct/  or 
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(2)  the  labels  on  a  line  do  not  agree;  In  this  case 
heuristics  are  Invoked  to  settle  the  Issue  In  favor  of  one  or 
the  other  of  the  labels. 

As  examples  of  these  heuristics,  Guzman  originally 
linked  regions  If  either  end  of  a  line  were  marked  with  a 
linking  label.  Later  he  added  a  system  using  "weak"  and 
"strong"  links  to  allow  more  subtle  weighting  of 
possibilities  and/  still  later/  he  added  a  link  Inhibition 
feature  which  provided  evidence  against  linking  certain 
regions.  Rattner  (Rattner  1970)  worked  out  various 
extensions  to  Guzman's  work  along  these  lines. 

As  It  turned  out,  the  link  Inhibition  feature  proved  to 
be  a  much  more  powerful  method  than  the  previous  methods  he 
had  tried.  Basically  this  Is  because  the  link  Inhibition 
technique  was  less  local  than  the  previous  links  had  been. 

The  assignment  of  a  link  Inhibition  between  two  regions  has 
consequences  for  every  line  which  separates  the  two  regions, 
unlike  the  linking  mark  which  only  serves  as  one  piece  of 
evidence  In  favor  of  linking  two  regions.  In  terms  of  my 
program,  the  program  using  links  only  Is  very  much  like  what 
my  program  would  be  If  1  divided  my  labels  up  Into  those 
which  had  PLUS  (convex)  marks  and  all  the  rest  (assume  that 
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there  are  no  shadows).  The  H nk  Inhibition  labels  would  he 
those  which  have  an  arrow  on  the  line  segment/  such  as 
occluding  edgeS/  crackS/  etc.  The  only  strong  evidence  for 
linking  regions  comes  from  ARROW  and  FORK  junctions,  and  of 
these  the  ARROW  junctions  are  the  more  Important/  since 
(ignoring  shadows  and  separable  PIUS  edges)  every  ARROW 
labeling  links  the  two  regions  which  bound  the  shaft  of  the 
ARROW,  In  contrast/  there  are  a  number  of  FORK  junctions 
which  have  non-linking  lines  (see  figure  9.2). 

But  If  link  inhibitions  are  used  there  Is  considerable 
evidence  In  ARROW/  T/  X/  and  K  junctions;  In  fact  Freuder 
has  shown  that  If  only  link  Inhibitions  are  used/  the  program 
works  just  about  as  well  as  Guzman's  full  program. 

There  are  numerous  problems  with  Guzman's  approach. 
First/  his  system  simply  does  not  work  very  well;  for 
carefully  chosen  scenes  It  will  find  the  correct  results/  but 
the  program  Is  very  easy  to  fool.  As  Winston  showed  (Winston 
1968)  Guzman's  program  falls  badly  on  scenes  with  holeS/  and 
obviously  the  program  Is  worthless  for  scenes  with  shadows. 

If  I  map  my  labelings  Into  Guzman's  binary  scheme  there  are 
examples  of  virtually  every  possible  labeling  for  each 
junction  type  within  my  data  base.  Thus  It  becomes  obvious 
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that  Guzman's  ;aba1s  are  simply  the  most  probable 
combinations  of  links  for  scenes  without  shadows.  As  such, 
his  program  really  has  very  little  understanding  of  the  world 
(see  Winston  1972a/  1972b). 

Second/  Guzman's  approach  Is  difficult  to  extend.  This 
is  due  to  the  use  of  only  one  labeling  for  each  junction  and 
consequent  heavy  dependence  on  special  purpose  heuristics/ 
and  due  also  to  the  fact  that  virtually  all  the  linking 
Information  for  a  line  comes  only  from  the  two  junctions  at 
the  ends  of  the  line.  There  is  no  systematic  way  to  use  any 
information  except  locally.  (The  only  exceptions  are 
Guzman's  use  of  matched  Ts/  the  link  inhibition  information/ 
and  regions  which  meet  along  more  than  one  edge.)  As  an 
example/  Orban's  extension  of  Guzman's  program  to  include 
shadows  (Orban  1970)  depends  exclusively  on  the  observation 
that  shadows  frequently  have  chained  L  and  X  junctions.  But 
despite  the  fact  that  Orban's  program  does  have  a  slightly 
greater  understanding  of  the  meani.g  that  scene  features  can 
have/  it  is  not  a  systematic  extension.  Like  almost  all  the 
extensions  suggested  for  Guzman's  work/  it  Isa  patchwork 
method:  to  handle  a  new  distinction,  pick  a  few  common 

features  that  display  the  distinction  and  then  adjust  the 
rest  of  the  program  to  avoid  making  disastrous  errors. 
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Third/  this  approach  leaves  a  great  deal  unexplained. 
Certainly  there  Is  a  great  deal  more  to  understanding  a  scene 
than  simply  being  able  to  connect  the  regions  Into  bodies. 

So  far  I  have  been  dealing  with  the  ways  In  which 
Guzman's  approach  was  deficient/  but  It  has  strong  features 
as  well.  Guzman  was  the  first  person/  to  my  knowledge/  to 
get  away  from  the  Idea  of  storing  descriptions  of  particular 
objects  and  trying  to  match  these  descriptions  to  a  given 
scene.  Roberts  (Roberts  1963)  had  used  this  method  and  In 
fact  others  continued  to  do  even  less  sophisticated  template 
matching  of  sorts  well  after  Guzman  published  his  work.  In 
contrast/  Guzman's  method  works  for  arbitrary  scenes 
containing  trihedral  vertices  and  gives  some  answer  for  any 
scene  presented  to  It.  Perhaps  the  most  appealing  feature  of 
SEE  was  Its  simplicity  and  clarity;  there  are  no 
tranformatlons/  coordinates/  or  hidden  llnes/  and  In  fact 
only  topology  Is  used.  Guzman's  great  Insight  was  that  by 
describing  the  physical  characteristics  of  a  relatively  small 
number  of  local  features/  one  can  use  simple  decision 
procedures  to  derive  much  less  local  facts  about  arbitrary/ 
unf ami  1 lar  scenes. 
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Guzman's  work  was  also  Instrumental  In  Initiating  two 
fruitful  lines  of  research  which  are  still  active.  This 
paper  is  along  the  line  defined  by  Huffman  and  Clowes 
(Huffman  1971#  Clowes  1971).  The  other  line  Is  the  work  on 
heterarchy  (For  excellent  discussions  of  both  Guzman  and 
heterarchy  see  Winston  1972a  or  1972b#  and  Minsky  &  Papert 
1972). 

9.2  WORK  AFTER  GUZMAN;  HUFFMAN  &  CLOWES 

Huffman  was  motivated  partly  by  his  obs:rvatlon  of  the 
lack  of  semantic  content  In  Guzman's  program  to  suggest  a 
richer  set  of  labels  than  link  and  do-not-link,  (Whether 
Clowes  came  upon  the  same  Ideas  Independent  of  Guzman  or  not 
I  do  not  know.)  Clearly  both  were  Influenced  by  Guzman's 
"grammatical"  approach  to  scene  processing.  Their  great 
Insight  was  that  by  describing  edges  more  precisely  one  could 
use  definite  rules  rather  than  probabilistically  based 
heuristics  to  choose  scene  Interpretations.  Moreover  they 
showed  that  one  could  even  say  with  some  assurance  that 
certain  line  drawing  could  not  even  correspond  to  real 
physical  scenes;  compare  this  with  the  fact  that  Guzman's 
program  rather  blindly  returns  some  decomposition  Into  bodies 
for  any  line  drawing#  and  you  will  get  some  Idea  of  the 


e 


SECTION  q. 2  297 

Increase  In  understanding  Implicit  In  Huffman's  and  Clowes' 
work. 

Both  Huffman  and  Clowes  also  worked  with  a  construction 
for  representing  region  orientations  called  the  dual  graph 
which  Influenced  my  thinking  on  region  orientations  (Huffman 
1971#  Clowes  et  al  1971).  Unfortunately#  there  Is  no  neat 
way  that  1  could  see  to  Integrate  the  dual  graph  Into  a 
labeling  scheme.  In  any  case#  I  owe  Huffman  and  Clowes  a 
considerable  debt. 

9.3  AN  ACCOUNT  OF  MY  EFFORTS 

When  Dowson  and  I  began  working  In  this  area#  we 
envisioned  a  tree  searching  program  which  would  attempt  to 
assign  labelings  from  a  reasonably  small  set  (like  those  of 
Huffman  and  Clowes)  to  a  line  drawing.  Dowson  came  up  with  a 
set  of  junctions  Involving  cracks#  and  1  generated  a  list  of 
shadow  junctions  (Dowson  &  Waltz  1971).  Dowson  then 
developed  VIRGIN#  a  tree  search  type  labeling  program  (Dowso- 
1971b)  to  apply  this  knowledge  to  real  scenes.  He 
Immediately  ran  Into  serious  problems#  since  even  the 
simplest  scenes  required  huge  amounts  of  computer  space#  and 
the  program  ended  up  with  many  possible  labelings  for  each 


scene.  Most  of  these  labelings  only  differed  by  one  or  two 
line  labels#  but  each  of  which  took  a  considerable  amount  of 
time  to  produce.  It  did  not  become  obvious  to  me  until 
somewhat  later  that  tree  search  was  the  wrong  model  for  this 
problem. 

In  my  proposal  for  this  work  (Waltz  1971)  I  suggested  a 
rather  heterarchical  model  for  labeling  line  drawings.  At 
this  time  I  had  already  noted  that  by  beginning  with  the 
scene/background  boundary  I  could  cut  down  the  search  space 
consider;**'!) .  ■'wd  I  listed  a  number  of  rules  (related  to  the 
selection  rules  and  region  illumination  types)  which  I 
thought  could  further  speed  up  and  Increase  the  power  of  a 
program.  I  also  showed  that  region  or ientatlor-  could  be 
handled  easily  If  1  restricted  the  universe  of  objects  to 
include  only  those  with  right-angle  edges. 

My  major  breakthrough  came  when  I  saw  that  the  region 
orientations  could  be  Included  as  part  of  the  edge  labels# 
and  then  saw  that  I  could  also  subdivide  each  edge  type  Into 
several  types  according  to  the  way  that  each  edge  could  be 
decomposed.  This  Idea  was  first  suggested  to  me  by  Freuder 
(see  Freuder  1971a)  nearly  a  year  before  I  used  it. 
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The  last  pieces  fell  Into  place  when  I  made  the  decision 
to  try  using  a  filtering  program  before  doing  a  tree  search, 
based  on  my  observations  of  Dowson's  difficulties.  Since  the 
set  of  labelings  1  now  had  was  far  larger  than  the  set  which 
had  clogged  his  program,  1  felt  that  1  needed  such  a  program 
to  clear  away  the  clutter  of  unneeded  labelings  and  make  tree 
searching  feasible.  1  was  genuinely  surprised  when  the 
filtering  program  returned  unique  labelings  for  most  of  the 
junctions  in  the  first  scenes  1  gave  to  it.  From  here  on  my 
work  followed  directly  from  the  success  of  the  combination  of 
this  filtering  program  and  the  much  enlarged  junction 
labeling  sets.  I  think  it  is  noteworthy  that  this  workts 
the  direct  result  of  my  interaction  with  the  program,  as 
opposed  to  being  the  result  of  a  system  I  worked  out  first  by 
hand  and  only  then  implemented  in  a  program. 


There  is  one  lesson  which  I  think  is  Important,  perhaps 
more  important  than  any  other  in  terms  of  the  ways  it  might 
aid  future  research.  For  a  long  time  after  1  had  found  the 
ways  of  describing  region  Illuminations  and  edge 
decompositions,  1  tried  to  find  a  clever  way  to  collapse  the 
large  set  of  line  labels  these  distinctions  implied  into  a 
smaller  and  more  manageable  set  which  would  retain  all  the 
"essential"  distinctions,  whatever  they  were.  Frustrated  in 
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this  attempt  for  quite  a  white/  I  finally  decided  to  go  ahead 
and  include  every  possible  labeling  in  the  program,  even 
though  this  premised  to  involve  a  good  deal  of  typing.  I 
hoped  that  when  I  ran  the  program  certain  regularities  would 
appear/  i.e.  that  when  the  program  found  a  particular 
labeling  for  a  junction  It  would  always  find  another  as  welt/ 
so  that  the  two  labelings  could  oe  collapsed  into  one  new  one 
with  no  loss  of  Information.  Of  course/  as  it  turned  out/  It 
was  the  fact  that  I  had  made  such  precise  distinctions  that 
allowed  the  program  to  find  unique  labelings.  The  moral  of 
this  is  that  one  should  not  be  afraid  of  semi -inf Ini t les;  a 
large  number  of  simple  facts  may  be  needed  to  represent  what 
can  be  deduced  by  computation  using  a  few  general  ideas. 

It  also  seems  logical  that.  If  anything/  people  s-e  able 
to  make  much  finer  distinctions  than  I  was  considering/  and 
that  these  distinctions  had  value  for  perception.  For 
example/  people  can  distinguish  between  obtuse  or  "blunt" 
edges  (such  as  those  of  a  regular  dodecahedron)/  right  angle 
'dges  (such  as  those  of  a  cube),  and  acute  or  "sharp"  edges 
(such  as  those  of  a  regular  tetrahedron). 
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Finally#  I  do  not  see  any  reason  to  suppose  that  we  i 

■I 

should  be  able  to  get  along  with  distinctions  on  the  order  of  I 

one  or  two  hundred#  any  more  than  a  language  program  wl th  a  i 

vocabulary  of  this  size  could  comprehend  or  express  anythin^  d 

2 
$ 

very  Interesting.  But  by  the  same  token#  It  may  be  that  a 
vision  system  does  not  have  to  be  too  large  for  available 
computers  In  order  to  reach  a  point  of  diminishing  returns# 
just  as  an  Increase  In  vocabulary  beyond  10# 000  words  would 
probably  not  add  much  to  a  language  program's  (or  a  person's) 
abl 1 1 ty. 
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